Notification System

A notification system is the shared plumbing many products need: given an event ("your order shipped"), deliver a message to the right users across the right channels — push, email, SMS, in-app — reliably. It's less about one clever trick and more about a clean pipeline that fans out, respects preferences, integrates flaky external providers, and never drops or duplicates messages. It's a direct application of Managing Long-Running Tasks.

1. Requirements

Functional:

• Accept a notification request (event + target audience + content).
• Deliver across channels: push (APNs/FCM), email, SMS, in-app.
• Respect user preferences (opt-outs, quiet hours, channel choice).

Non-functional:

• Reliable — at-least-once delivery; retries on provider failure; nothing silently lost.
• Scale — bursts of millions (a broadcast to all users).
• No duplicates — a user shouldn't get the same notification twice.
• Timely — delivered promptly, but async (not in the triggering request's path).

2. The pipeline

Never send notifications inline in the triggering request. Accept the request, queue it, and process asynchronously:

event → ingest API → enqueue →
  resolve recipients → apply preferences/filters →
  fan out per channel → hand to channel workers →
  per-channel provider (APNs/FCM/email/SMS) → record status

A queue between stages absorbs bursts (a "notify everyone" broadcast) and decouples the fast ingest from slow external sends.

3. Fan-out and preferences

• Resolve audience: one logical event may target one user or millions (broadcast). Expand it to concrete recipients. A huge broadcast fans out via the queue so no single step blocks.
• Apply preferences: before sending, filter by each user's settings — opted out of marketing? quiet hours? prefers email over push? This is the policy layer, consulted per recipient. Honoring opt-outs is also a compliance requirement, not just UX.
• Per-channel split: a recipient may get the same notification on multiple channels, or a preferred one; split into per-channel send tasks.

4. Channel workers and providers

Each channel has its own worker pool and talks to an external provider (APNs/FCM for push, an email provider, an SMS gateway). Isolating channels means a slow email provider doesn't stall push, and each can scale and rate-limit independently. Workers:

• Call the provider, handle its specific errors and rate limits.
• Retry with exponential backoff on transient failure; dead-letter after a cap.
• Record delivery status (sent / failed / bounced) for observability.

5. Reliability: at-least-once + dedup

Delivery is at-least-once (retries can duplicate), so prevent duplicates with an idempotency/dedup key per (notification, user, channel): record what's been sent and skip a repeat. This is the same at-least-once-plus-idempotency pattern as the Job Scheduler — the honest guarantee is "at least once, deduplicated," not magic exactly-once.

Persist each notification's state so a crash mid-pipeline resumes rather than loses or re-blasts.

6. Rate control and abuse

• Per-user rate limiting / batching: don't flood one user — collapse many events into a digest, cap per hour (Rate Limiter).
• Provider rate limits: respect APNs/SMS throughput caps; the per-channel queue smooths sends to the allowed rate.
• Priority: a 2FA code must jump ahead of a marketing blast — priority queues or separate lanes per urgency.

7. Deep dives

• Templating: content is rendered from templates with per-user data and localization, kept separate from delivery.
• Broadcast scale: "notify all users" is fanned out through the queue in chunks so it drains steadily instead of toppling providers.
• Feedback loop: handle bounces/unsubscribes/invalid tokens by updating preferences so you stop sending to dead endpoints.
• In-app channel: stored and pushed via a real-time channel; others go through external providers.

What to emphasise

Describe the async fan-out pipeline and the at-least-once + dedup guarantee:

"I'd make it an async pipeline: an event is accepted and queued, then I resolve the audience, apply each user's preferences and quiet hours, and fan out into per-channel send tasks. Each channel — push, email, SMS — has its own worker pool and provider integration, isolated so a slow provider on one channel doesn't stall the others, with exponential-backoff retries and a dead-letter after a cap. Delivery is at-least-once, so I dedup on a per-notification-per-user-per-channel key to avoid double-sends. Queues absorb broadcast bursts, per-user rate limiting and batching prevent flooding, and urgent messages like 2FA codes use a priority lane."

Next up: Stock Trading — a low-latency matching engine.