Rate Limiter

A rate limiter restricts how many requests a client can make in a window — "100 requests per minute per user." It protects your system from abuse, runaway clients, and accidental overload, and it's a deceptively deep little design: the algorithms have real trade-offs, and making it correct across a fleet of servers forces you to think about shared state.

1. Requirements

Functional:

• Allow a request if the client is under its limit; reject (HTTP 429 Too Many Requests) if over.
• Limits are per-identity (per user, API key, or IP) and configurable.

Non-functional:

• Low latency — it runs on every request, so it must add well under a millisecond.
• Correct across all servers — a user hitting different servers must still share one limit.
• Fail open, not closed — if the limiter itself is down, prefer letting traffic through over blocking everyone (usually; a strict system may choose otherwise).

2. Where it lives

Rate limiting belongs at the edge, before requests reach your business logic — typically in the API Gateway or a middleware in front of your services. That way an over-limit request is rejected cheaply, before doing any real work.

3. The algorithms

This is the heart of the problem. Each trades accuracy against memory and smoothness:

Fixed window. Count requests per fixed clock window ("per minute"); reset the counter at the boundary. Dead simple — one counter per user. The flaw: a client can send the full limit at 12:00:59 and again at 12:01:00 — double the limit across a one-second span, because the window resets sharply.

Sliding window log. Store the timestamp of every request; count how many fall in the last 60 seconds from now. Perfectly accurate and smooth, but stores a timestamp per request — memory grows with traffic.

Sliding window counter. A pragmatic middle ground: keep per-window counts and weight the previous window by how much of it still overlaps the rolling window. Smooths the fixed-window boundary problem with roughly one counter's worth of memory. A common production choice.

Token bucket. A bucket holds up to N tokens and refills at a steady rate; each request takes a token, and an empty bucket means reject. Elegant because it allows bursts (up to the bucket size) while capping the long-run rate — which matches how you usually want APIs to behave. Store just two numbers per client (token count, last-refill time).

Token bucket is the usual default — tiny state, controlled bursts, simple to reason about.

4. Making it correct across servers

The trap: if each server keeps its own counter, a user spread across 3 servers gets 3× their limit. The limit must live in shared state — and Redis is the natural home: one counter/bucket per client, visible to every server, updated with an atomic operation so concurrent requests can't race.

request → server → Redis: atomically check-and-update this user's bucket
                        → under limit? allow : 429

The atomicity matters — INCR (or a small Lua script for token bucket) does the read-modify-write in one step, so two simultaneous requests can't both slip past the limit. Set a TTL on the key so idle clients clean themselves up.

5. Deep dives

• Latency: the Redis round trip is sub-millisecond on the same network — acceptable on the hot path. For extreme cases, a local in-memory limiter per server as a coarse first gate, backed by Redis for the precise shared limit.
• Fail open: if Redis is unreachable, allow the request (log it) rather than reject everyone — a broken limiter shouldn't become an outage.
• Response: return 429 with a Retry-After header so good clients back off politely.
• Tiers: different limits per plan (free vs. paid) is just a different limit value keyed by the client.
• Distributed accuracy vs. cost: perfectly global limits cost a shared round trip per request; many systems accept slightly fuzzy limits (per-node buckets summing to roughly the target) to save it.

What to emphasise

Name the algorithm and why, and insist on shared state:

"I'd put a token-bucket limiter in the API gateway — it caps the sustained rate but allows a controlled burst, which is how APIs should behave, and it's just two numbers per client. The state has to be shared or a user on three servers gets triple the limit, so I'd keep the buckets in Redis and update them atomically with a small Lua script. Over-limit returns 429 with Retry-After, and if Redis is down I fail open rather than block everyone."

Next up: Chat App — real-time messaging at scale.