Question Breakdowns

News Aggregator

A worked system design: pull articles from many sources, deduplicate near-identical stories, and serve a ranked, personalized feed (think Google News). Ingestion, clustering, and ranking.

A news aggregator pulls articles from thousands of sources, groups the many reports of the same story together, and serves users a fresh, ranked feed. The distinctive challenges are ingestion from heterogeneous feeds, deduplication/clustering (ten outlets covering one event should appear as one story), and ranking for freshness and relevance. It combines ideas from the Web Crawler, Post Search, and News Feed.

1. Requirements

Functional:

  • Ingest articles from many sources (RSS, APIs, crawling).
  • Group articles covering the same story into one cluster.
  • Serve a ranked feed; optionally personalized by topic/interest.

Non-functional:

  • Fresh — breaking news appears within minutes.
  • Scale — thousands of sources, high article volume, many readers.
  • Read-heavy on the feed; ingestion is continuous background work.
  • Dedup quality matters — duplicated stories ruin the feed.

2. Ingestion

A fleet of ingestion workers polls each source on a schedule (RSS/APIs) or crawls it (Web Crawler techniques). This is a scheduled, pull-based pipeline:

sources → poll/crawl (per-source schedule) → parse article (title, body, time, source)
        → dedup check → store article + assign to a story cluster → index

Poll frequency tracks a source's update rate (a wire service often, a blog rarely) — the same freshness-driven scheduling as a crawler's frontier. Normalize each article into a common shape regardless of source format.

3. The interesting part: clustering duplicate stories

Ten outlets report the same earthquake; the feed should show one story (with sources), not ten rows. This is near-duplicate detection / clustering:

  • Near-dup detection: fingerprint article content (e.g. SimHash/MinHash, Big Data Structures) so near-identical rewrites cluster even when wording differs slightly.
  • Semantic clustering: embed articles into vectors and group by closeness, so different-worded reports of the same event cluster by meaning, not just text overlap.

Each incoming article is compared to recent clusters: close enough → join that story; otherwise → start a new one. Only compare against recent clusters (a time window) so you're not diffing against the whole archive.

new article → fingerprint/embed → match recent clusters?
   yes → attach to existing story
   no  → create new story cluster

4. Serving the feed

The feed reads from a store of story clusters, each with its representative article, sources, and timestamp. It's read-heavy, so cache the ranked feed and refresh it as new stories/clusters form. Search over articles uses an inverted index (Post Search).

5. Ranking

Rank stories by a blend of freshness (recency decays fast for news), importance (how many sources picked it up — a big story is covered widely), and personalization (match to the user's topics). As with the News Feed, keep mechanism (clustering + candidate stories) separate from policy (the ranking score). Personalization filters/reweights the shared candidate set per user.

6. Deep dives

  • Freshness vs. cost: clustering every article against recent ones is work — bound it with a time window and coarse pre-filtering (candidate clusters by topic) before expensive comparison.
  • Topic classification: tag each story (politics, sports) for personalization and section feeds — a classifier in the pipeline.
  • Source trust / spam: weight reputable sources, filter spam/content farms.
  • Eventual consistency: a new article taking a minute to join its cluster or appear is fine — it's an AP feed.

What to emphasise

Lead with ingestion, make clustering the centerpiece, then ranking:

"Ingestion is a scheduled pull/crawl across thousands of sources, poll frequency matching each source's update rate, normalizing articles into a common shape. The distinctive part is clustering: many outlets cover one event, so I fingerprint articles with SimHash or embed them as vectors and match each new article against recent clusters, joining an existing story or starting a new one — comparing only within a time window to bound the cost. The feed reads from story clusters, cached and read-heavy, ranked by a blend of freshness, how widely a story is covered, and personalization, keeping clustering separate from the ranking policy."

Next up: Price Tracker — watching values and alerting on change.