Ad Click Aggregator

An ad click aggregator ingests a massive stream of click events and turns them into aggregated counts — clicks per ad, per campaign, per minute/hour — that power advertiser dashboards and, crucially, billing. Unlike Top-K Trending, approximation is not acceptable here: advertisers are charged per click, so the counts must be accurate and auditable. That one requirement — accuracy under a firehose — drives the whole design. It's the canonical stream-processing / scaling-writes problem.

1. Requirements

Functional:

• Ingest click events (ad id, campaign, timestamp, user/context).
• Serve aggregated counts over time windows (per-minute, hourly, daily) sliced by ad/campaign.
• Power both near-real-time dashboards and end-of-period billing.

Non-functional:

• Firehose ingest — millions of clicks/sec, bursty.
• Accurate — counts feed billing; no lost or double-counted clicks.
• Low-latency reads — dashboards query precomputed aggregates instantly.
• Near-real-time — a few seconds/minutes of lag is fine; the raw stream need not be queryable live.

2. Don't count synchronously

Writing a row and incrementing a counter per click, synchronously, at millions/sec would crush any database and couple ingest latency to aggregation. The pattern is ingest fast, aggregate async (Scaling Writes):

click → lightweight collector → append to durable log (Kafka)
      → stream processor consumes → windowed aggregation → write rollups
dashboard/billing ← query precomputed rollups (small, fast)

The collector does almost nothing but append the event to a durable, partitioned log — Kafka — which absorbs bursts and becomes the source of truth. Aggregation happens downstream at its own pace.

3. Stream aggregation

A stream processor — Flink is purpose-built for this — consumes the event log and maintains windowed aggregates: tumbling 1-minute buckets of (ad_id, campaign) → count. As each minute closes, it emits the rollup to a serving store (a database or OLAP store) that dashboards query. Pre-aggregating by minute means a "last 7 days" query sums ~10k minute-rows, not billions of raw clicks.

events ─► Flink: group by (ad, minute) ─► emit per-minute counts ─► rollup table

4. The hard part: exactly-once accuracy

Because this feeds billing, you must not lose clicks or double-count them. Two defenses:

• Idempotency / dedup: each click carries a unique id; the pipeline dedups on it so a retried or replayed event is counted once. (Networks retry; consumers restart and reprocess — without dedup you over-bill.)
• Exactly-once stream processing: Flink's checkpointing plus Kafka offsets give exactly-once semantics — on failure it rewinds to the last checkpoint and reprocesses without double-counting committed output.

The durable log makes this possible: if a downstream step fails, you replay from the log and recompute — nothing is lost because the raw events are retained.

5. Reconciliation: fast vs. correct

A clean trick is a two-path (lambda-style) design:

• Speed path: approximate/near-real-time counts for the live dashboard (a few seconds behind).
• Batch path: a periodic exact recomputation over the raw log that corrects the numbers for billing.

Dashboards get immediacy; billing gets a slower, authoritative, auditable total that reconciles any drift. Many systems run both over the same Kafka log.

6. Deep dives

• Hot keys: a viral ad skews one partition — pre-aggregate locally per consumer before the global rollup to spread load.
• Late / out-of-order events: a click arriving late (mobile offline) must land in the right time window — use event-time windows with a watermark/grace period (Flink handles this).
• Fraud: filter bot clicks before billing — a separate scoring step on the stream.
• Storage: raw events tier to cheap cold storage after the retention window; rollups stay hot.

What to emphasise

Contrast it with approximate counting, then name exactly-once:

"Unlike trending, this feeds billing, so it must be accurate — no lost or double-counted clicks. I'd ingest clicks into a durable partitioned log like Kafka, which absorbs the firehose and is the source of truth, then aggregate asynchronously in a stream processor like Flink into per-minute rollups that dashboards query cheaply. Accuracy comes from idempotent dedup on a click id plus exactly-once processing via checkpointing and offsets, and because raw events are retained I can replay to recompute. For billing I'd run a batch reconciliation over the log to correct the fast approximate path, and use event-time windows so late clicks land in the right bucket."

Next up: Post Search — full-text search over a growing corpus.