Key Technologies

Flink

Stream processing for data in motion — continuous computation over event streams with windowing and stateful aggregation, and how it differs from batch jobs and from Kafka itself.

Apache Flink is a stream-processing engine: it runs continuous computations over unbounded streams of events, updating results in real time as data arrives. Where a batch job runs once over a fixed dataset, Flink runs forever over a never-ending flow — aggregating, joining, and transforming events the moment they happen. It's the tool for "compute this as the data streams in," not "compute this nightly."

What it's for

Classic jobs where the answer must be live:

  • Real-time aggregation — ad clicks per minute, live dashboards, running counts and sums.
  • Windowed analytics — "orders in the last 5 minutes," "unique users this hour."
  • Streaming ETL — reshape and enrich events on the way from a source (Kafka) to a sink (a database, a search index).
  • Pattern / anomaly detection — watch a stream for sequences that signal fraud or an outage.
[ Kafka: raw clicks ] ─► [ Flink: window + aggregate per minute ] ─► [ serving store ]

Windowing and state

Two ideas make stream processing more than "a loop over events":

  • Windows. You can't sum an infinite stream, so you compute over slices of time — tumbling (fixed, non-overlapping: each minute), sliding (overlapping), or session windows (grouped by activity gaps). "Clicks per minute" is a tumbling window.
  • Managed state. Flink keeps large, fault-tolerant state (the running counts, the join tables) and checkpoints it, so if a node fails the computation resumes exactly once without double-counting. That exactly-once guarantee over big state is Flink's hard-won strength.

It also distinguishes event time (when something actually happened) from processing time (when Flink saw it), and handles late-arriving events — important when network delays mean events show up out of order.

These get muddled:

  • Kafka transports and stores the stream (the pipe/log).
  • Flink computes over the stream (the processor).
  • A batch job (Spark, a nightly SQL job) computes over data at rest, on a schedule.

A common shape is Kafka → Flink → store: Kafka absorbs the firehose, Flink aggregates it continuously, the results land somewhere you can read cheaply. (Lighter needs can use Kafka Streams or a simple consumer instead of a full Flink cluster.)

What to watch

  • Operational weight. A Flink cluster with stateful, checkpointed jobs is serious infrastructure. For simple per-event work, a plain Kafka consumer is far less to run.
  • It's compute, not storage. Results must be written somewhere to be served; Flink holds working state, not your source of truth.

What to say in an interview

Bring it up when you need live aggregates over a stream, downstream of Kafka:

"The ad-click dashboard needs per-minute counts in real time, not a nightly batch. I'd stream clicks through Kafka and run a Flink job that aggregates them in tumbling one-minute windows, using event time so late clicks land in the right bucket, and write the rollups to a store the dashboard reads. Flink's checkpointing gives exactly-once so I don't double-count. For something this simple I'd consider Kafka Streams to avoid running Flink."

Next up: ZooKeeper — coordination for distributed systems.