Metrics Monitoring

A metrics monitoring system collects operational metrics (CPU, request latency, error rates) from thousands of services, stores them, and powers dashboards and alerts. It's the operational cousin of the Activity Tracker: a write-heavy flood of time-stamped numbers, queried by time range, where the storage model (Time-Series Databases) and the alerting path are the whole game.

1. Requirements

Functional:

• Ingest metrics (name, labels/tags, value, timestamp) from many sources.
• Query metrics over time ranges, aggregated and sliced by label.
• Evaluate alert rules and fire notifications when thresholds are breached.

Non-functional:

• Write-heavy, high-cardinality — millions of data points/sec; many metric+label combinations.
• Fast range queries — dashboards aggregate over time windows.
• Scalable retention — raw data is huge; old data rarely needs full resolution.
• Timely alerts — breaches detected within seconds.

2. The data model: labeled time-series

A metric is a named series identified by its labels: http_requests{service="api", status="500"}. Each unique label combination is a distinct series of (timestamp, value) points. This is classic time-series data — append-only, time-ordered, read by range:

series key = metric name + label set
  http_requests{service=api,status=500} → [(t1,v1),(t2,v2),...]

The defining scaling concern is cardinality: every distinct label combination is a new series, and high-cardinality labels (like user_id) explode the series count and memory. A core design rule is to keep label cardinality bounded.

3. Ingest: pull or push

Two models, both common:

• Pull (scrape): the system periodically scrapes a /metrics endpoint on each service (Prometheus's model). The monitor controls load and knows what's up (a failed scrape is itself a signal).
• Push: services/agents push metrics to an ingest endpoint (Datadog's model). Better for short-lived jobs and across network boundaries.

Either way, ingested points append to the time-series store. At high volume, buffer through a queue and write in batches — appends dominate, so this is a Scaling Writes problem; partition series across nodes by series key.

4. Storage: TSDB with rollups

A purpose-built time-series database is the right store: time-partitioned, heavy compression of sequential points (adjacent timestamps/values compress enormously), and fast range scans. Crucially, use downsampling + retention:

raw (seconds) ──roll up──► 1-min ──roll up──► 1-hour
keep days      keep weeks        keep months/years

A dashboard showing "last 90 days" reads coarse rollups (hourly), not billions of raw points; "last 5 minutes" reads raw. Old raw data is dropped on a retention policy — you keep the summaries, not every original sample. This keeps storage and query cost bounded.

5. Querying and dashboards

Queries aggregate a series over a window and group by label: "p99 latency per service over the last hour." The engine picks the right resolution (raw vs. rollup) for the range, scans the matching series, and aggregates. Results feed dashboards, which are read-heavy and cacheable for common views.

6. Alerting

Alert rules are continuously-evaluated queries: "fire if error rate > 5% for 5 minutes." An alert evaluator runs each rule on a schedule against recent data; on breach it emits an alert to a Notification System. Essentials:

• For-duration / hysteresis: require the condition to hold for a period so a one-sample blip doesn't page someone.
• Deduplication & grouping: collapse a storm of related alerts (a whole cluster down) into one, and don't re-page while an alert is already firing.
• Routing/escalation: send to the right on-call, escalate if unacknowledged.

7. Deep dives

• Cardinality control: reject or limit high-cardinality labels; they're the #1 way these systems fall over.
• Hot recent data: keep the last few hours in memory/fast storage (most queries and all alerts are recent); age the rest to disk/cold tiers.
• Late/out-of-order points: agents retrying after a blip send late data — accept into the right time bucket (event-time), like the Ad Click Aggregator.
• Federation: at huge scale, regional monitors aggregate up to a global view.

What to emphasise

Name the time-series model, cardinality, rollups, and the alerting loop:

"It's labeled time-series data — write-heavy, append-only, read by range — so I'd store it in a time-series database with time partitioning and heavy compression, buffering high-volume ingest through a queue and partitioning series by key. The defining concern is cardinality: every label combination is a new series, so I'd bound high-cardinality labels. Rather than keep everything at full resolution, I downsample into coarser rollups with a retention policy, so long-range dashboard queries read hourly rollups and only recent queries hit raw data. Alerting is a separate evaluator running rule queries on a schedule against recent data, firing to a notification system — with a for-duration so blips don't page, plus dedup and grouping so an outage is one alert, not a thousand."

Next up: Real-time Game — low-latency shared state for many players.