Kafka

Kafka is a distributed, durable, append-only log. Producers append events to the end; consumers read forward at their own pace. That simple shape makes it the backbone for event-driven systems — a buffer that absorbs huge write bursts, a pipe that decouples services, and a replayable record of everything that happened.

The model

• Topic — a named stream of events ("clicks," "orders").
• Partition — a topic is split into partitions, each an ordered log. Partitions are the unit of parallelism and of ordering: events are ordered within a partition, not across them. The partition key (e.g. user_id) decides which partition an event lands in — so all of one user's events stay ordered.
• Consumer group — a set of consumers that share the work; each partition is read by one consumer in the group, so adding consumers (up to the partition count) scales throughput.
• Offset — each consumer tracks how far it has read. Because Kafka retains events (for days, or forever), a consumer can rewind and replay.

producers ─► [ topic: partition 0 | 1 | 2 ] ─► consumer group (one consumer per partition)
                     (ordered within each partition, retained on disk)

The three jobs it owns

1. Absorb a firehose. Writing millions of events/sec straight to a database melts it. Kafka takes them at log speed; consumers drain into the database at a sustainable rate (see Scaling Writes).
2. Decouple services. The order service publishes "order placed"; email, inventory and analytics each consume it independently. Add a new consumer without touching the producer. One event, many reactions.
3. Replayable event stream. Because events are retained, you can reprocess history — rebuild a search index, backfill a new feature, recover a downstream store — by replaying from an earlier offset.

Delivery semantics

Kafka is at-least-once by default: a consumer may see an event more than once (e.g. it crashed after processing but before committing its offset). So consumers must be idempotent — the same lesson as background jobs. Exactly-once is possible within Kafka with extra machinery, but designing idempotent consumers is the pragmatic path.

What to watch

• Ordering is per-partition only. If you need all of an entity's events in order, key them to the same partition.
• It's not a database. You consume forward; it's not for random point lookups. Pair it with a serving store.
• Operational weight. A Kafka cluster is real infrastructure. For modest async work, a job queue (Redis/BullMQ, SQS) is far simpler — reach for Kafka when you need high throughput, multiple independent consumers, or replay.

What to say in an interview

Reach for it to buffer, decouple, or replay — and know when a plain queue is enough:

"The click stream is a firehose, so I'd publish clicks to a Kafka topic partitioned by something that spreads evenly. A stream processor consumes and aggregates them, and analytics consumes the same topic independently. Retention lets me replay to rebuild the aggregates if the logic changes. For just sending emails off the request path, I'd use a simple queue, not Kafka."

Next up: Temporal — durable orchestration for long, multi-step workflows.