Managing Long-Running Tasks

Some work simply can't finish in the 100 milliseconds a user is willing to wait: sending email, transcoding a video, generating a report, calling a slow third-party API, running a nightly rollup. If you do it inside the request, the endpoint is slow and fragile — worse, an outside service being down makes your core feature fail (signup breaks because the email provider is down). The pattern is to get that work off the request path.

The core move: enqueue, respond, process later

Split "what the user must wait for" from "what can happen afterwards." The endpoint does the minimum, drops the rest on a queue, and responds. A separate worker process pulls from the queue and does the slow work.

POST /signup → create the user (must wait) → enqueue "send welcome email" → 201
worker       → pulls the job → sends the email (user already got their response)

Signup now succeeds instantly and still succeeds even if the email provider is down — the job just waits in the queue. The cost is that "done" now means "promised": design the UI for it ("Check your inbox in a moment").

Workers scale independently

Because the queue decouples producers (web servers) from consumers (workers), you scale them separately. A backlog building up means "add workers"; it doesn't touch your web tier. Run workers as their own service, not inside the web server, so a heavy job can't starve request handling.

Build retries in from the start

Jobs fail — a flaky API, a transient blip. The queue should retry with backoff (wait longer between attempts), and after a few failures move the job to a dead-letter queue where a human can inspect it, instead of either losing it or retrying forever. Two rules make this safe:

• Idempotency. A retried job may run twice. Sending an email twice is fine; charging a card twice is not — use idempotency keys and "done" markers.
• Pass ids, not whole objects. Enqueue { userId } and load fresh data in the worker; payloads go stale while they wait.

Scheduled and recurring work

For "every night at 2am" or "every 5 minutes," you need a scheduler. The trap: if you run a cron on every instance, three instances send three copies of the daily digest. The fix is a single scheduler or a queue with repeatable jobs, so each scheduled run is picked up exactly once regardless of how many instances exist.

When the task is genuinely long and stateful

A simple queue is perfect for "do this one thing soon." But some processes run for minutes, hours or days and have many steps with waits in between — an onboarding flow, a multi-day retry, a human-approval step. Hand-rolling that with queues and status columns gets fragile fast. Durable workflow engines (e.g. Temporal) are built for it: they persist the state of a long workflow, survive restarts, and resume exactly where they left off, handling retries and timeouts for you. Reach for one when the orchestration — not the individual task — is the hard part.

Choosing

What to say in an interview

Move slow or external work off the request, and make the jobs reliable:

"I'd do the minimum in the request and queue the rest. Signup creates the user, then enqueues the welcome email so an email outage can't fail signup. Workers run separately with retry + backoff and a dead-letter queue for jobs that keep failing; each job is idempotent since it'll be retried. The nightly report is a repeatable job so it runs exactly once across all instances. If the flow were long and multi-step I'd use a workflow engine instead of stitching queues together."

That completes Patterns. Next, the Key Technologies section puts names to the tools these patterns call for.