Web Crawler

A web crawler systematically fetches pages across the web, extracts their links to discover more pages, and stores the content for downstream use (a search index, training data). It's essentially a massive, distributed, breadth-first traversal of an unimaginably large graph. The design challenges are keeping the work queue sane, being polite to the sites you crawl, and not fetching the same thing twice. It's a pure long-running, distributed pipeline.

1. Requirements

Functional:

• Start from seed URLs; fetch each page; extract links; enqueue new URLs; repeat.
• Store fetched content for downstream processing (e.g. indexing).
• Re-crawl periodically to keep content fresh.

Non-functional:

• Scale — billions of pages; thousands of fetches/sec; runs continuously.
• Polite — respect robots.txt and don't hammer any single domain.
• No duplicate work — don't crawl the same URL (or identical content) repeatedly.
• Robust — handle timeouts, redirects, dead links, traps, without stalling.

2. The core loop

URL frontier (queue of URLs to fetch)
  → fetcher: download page (check robots.txt, rate-limit per domain)
  → parser: extract text + links
  → store content
  → dedup + filter links → enqueue new URLs back into the frontier

Everything hangs off the URL frontier — the queue of what to crawl next. Its design is the whole problem.

3. The URL frontier

The frontier isn't a plain FIFO queue; it must enforce two things at once:

• Priority: important/fresh pages first (homepages, frequently-changing sites).
• Politeness: never fire many concurrent requests at one domain, and space out requests to the same host.

The standard design is per-host queues behind the frontier: URLs are grouped by domain, and a scheduler pulls from host-queues such that each host is fetched no faster than its politeness delay, while many different hosts are fetched in parallel. This keeps throughput high globally while being gentle per site. A durable queue (Kafka or a database-backed queue) holds the frontier since it's enormous and must survive restarts.

4. Deduplication

Two kinds, both essential:

• URL dedup — have we already seen/queued this URL? With billions of URLs, an exact set is huge, so use a bloom filter (Big Data Structures) for a fast, memory-cheap "probably seen" check, backed by an authoritative store for confirmation. Normalise URLs first (strip fragments, sort query params) so trivially-different URLs collapse.
• Content dedup — different URLs often serve identical content (mirrors, print pages). Hash the page content and skip storing/indexing duplicates.

5. Politeness and traps

• robots.txt: fetch and cache each domain's rules; honour disallows and crawl-delay. Non-negotiable.
• Rate-limit per domain: the per-host queue scheduling enforces this.
• Crawler traps: infinite calendars, session-id URLs that generate endless unique links. Defend with max depth per site, URL-count caps per domain, and pattern detection.

6. Scaling the fetch

Fetching is I/O-bound and embarrassingly parallel — run a large fleet of stateless fetcher workers pulling from the frontier. Partition the frontier by domain (consistent hashing) so one host's politeness state lives on one worker and isn't violated by others. Content goes to blob storage; extracted metadata/links to a database; the index build is a separate downstream job feeding Elasticsearch.

7. Deep dives

• Freshness / re-crawl: schedule re-crawls by change frequency — news every hour, static pages rarely. A priority on the frontier, informed by observed change rate.
• Distributed coordination: avoid two workers crawling the same host; partition by host and use the dedup store as the shared "seen" state.
• Backpressure: if downstream storage/indexing lags, slow the fetchers — the durable frontier absorbs the gap.

What to emphasise

Center the frontier, politeness, and dedup — that's where the signal is:

"It's a distributed BFS of the web hung off a URL frontier. The frontier isn't a plain queue — it's organised into per-host queues so I can crawl many domains in parallel while rate-limiting and respecting robots.txt for each one, which is the politeness requirement. I dedup URLs with a bloom filter over normalised URLs since an exact set of billions is too big, and dedup content by hashing pages. Fetchers are a stateless parallel fleet pulling from a durable frontier, partitioned by host so politeness state stays consistent, and I defend against crawler traps with depth and count caps. Content lands in blob storage; indexing is a separate downstream pipeline."

Next up: Ad Click Aggregator — counting a firehose, accurately.