Job Scheduler

A job scheduler runs tasks at a specified future time or on a recurring schedule — "send this email in 2 hours," "run this report every night," "retry this payment in 10 minutes." It's "cron as a service," and the challenges are reliability-focused: finding what's due efficiently among millions of pending jobs, running each exactly once despite failures, and retrying sensibly. It builds on Managing Long-Running Tasks.

1. Requirements

Functional:

• Schedule a job to run once at time T, or on a recurring cron schedule.
• Execute due jobs by invoking a handler (call a service, send a message).
• Support cancellation and retries on failure.

Non-functional:

• Reliable — a scheduled job must run, even across server restarts; nothing silently dropped.
• Exactly-once-ish — don't run a job twice; or make execution idempotent.
• Scale — millions of pending jobs; many firing per second at peak.
• Timely — jobs fire close to their due time, not minutes late.

2. Storing jobs

Jobs live in a durable store (not memory — they must survive restarts), each with a run_at timestamp and status:

jobs ( id, run_at, payload, status, attempts, next_retry_at, schedule? )

The central query is "which jobs are due now?" — WHERE status='pending' AND run_at <= now(). An index on run_at makes this a cheap range scan over just the due jobs instead of scanning millions (Database Indexing). This indexed due-time lookup is the heart of the scheduler.

3. The dispatch loop

A dispatcher polls for due jobs and hands them to workers:

dispatcher: every tick → fetch due jobs (run_at <= now, pending)
          → claim each atomically (status → 'running')
          → enqueue onto a work queue
workers: consume queue → execute handler → mark done / schedule retry

Separating dispatch (deciding what's due) from execution (doing the work) via a queue lets you scale workers independently and absorb bursts when many jobs fire at once.

4. Exactly-once: the claim

The danger: two dispatcher instances (you run several for availability) both pick up the same due job and run it twice. Prevent it by atomically claiming a job before dispatching — a conditional update like UPDATE jobs SET status='running', owner=? WHERE id=? AND status='pending'. Only one dispatcher wins the row; the other sees 0 rows updated and skips it. This is the same contention trick used everywhere in this track.

Even so, treat execution as at-least-once and make handlers idempotent (via an idempotency key), because a worker can crash after running the job but before marking it done — on recovery the job looks un-run and reruns. Idempotency is the honest guarantee; "exactly once" is best-effort on top of it.

5. Retries and failure

• Failed job: increment attempts, set next_retry_at with exponential backoff, return it to pending. Cap attempts, then send to a dead-letter queue for inspection.
• Crashed worker: a claimed running job whose owner died must not be stuck forever — a visibility timeout / lease: if a running job isn't completed within its lease, it's reclaimed and retried.

6. Recurring jobs

A cron-style job, on completion, computes its next run_at from the schedule and re-inserts itself as pending. So recurrence is just "run, then schedule the next occurrence" — no special machinery.

7. Deep dives

• Timeliness vs. load: polling frequently is timely but costly; a short poll interval plus the run_at index balances it. For far-future jobs, a tiered design (coarse store for distant jobs, promoted to a near-term queue as they approach) avoids scanning everything.
• Scale the due-scan: shard jobs by id/tenant so each dispatcher owns a partition and scans only its slice.
• Hot second: millions due at exactly midnight — the queue between dispatch and execution absorbs the spike; workers drain it over the following moments.
• Managed option: Temporal provides durable timers and exactly-once-style workflow execution out of the box.

What to emphasise

Center the indexed due-time lookup, the atomic claim, and idempotent execution:

"Jobs are stored durably with a run_at and an index on it, so 'what's due now' is a cheap range scan, not a full scan of millions. A dispatcher polls for due jobs and atomically claims each with a conditional update so multiple dispatcher instances can't run the same job twice, then enqueues it for a separate pool of workers — decoupling dispatch from execution so I can scale workers and absorb bursts. Because a worker can crash after running but before acking, I treat execution as at-least-once with idempotent handlers and a lease that reclaims stuck jobs. Failures retry with exponential backoff and a dead-letter after a cap, and recurring jobs just re-schedule their next occurrence on completion."

Next up: News Aggregator — pulling, dedup, and ranking many sources.