Payment System

A payment system charges customers and records the movement of money. Here correctness isn't one requirement among many — it's the only thing that matters, because every bug is someone's real money. The design is dominated by three ideas: idempotency (never double-charge), a ledger as the immutable source of truth, and orchestrating a slow, unreliable external gateway reliably. It leans on Multi-step Processes and Dealing with Contention.

1. Requirements

Functional:

• Charge a customer for an amount (via card/gateway); record the result.
• Support refunds; expose payment status.
• Maintain an accurate, auditable record of every money movement.

Non-functional:

• Correct above all — never double-charge, never lose a payment, never show a wrong balance.
• Strongly consistent — money is not an eventually-consistent domain.
• Durable & auditable — every transaction permanently recorded and reconstructible.
• Resilient — handle gateway timeouts/failures without losing track of money.

2. Idempotency: the first principle

Networks retry. A user double-clicks "Pay"; a timeout makes the client resend; a worker retries after a crash. Without protection, each becomes a separate charge. Every payment request carries an idempotency key; the server records it and, on a repeat, returns the original result instead of charging again (Multi-step Processes):

charge(idempotency_key) →
  key seen?  → return the stored result (no new charge)
  key new?   → process once, store result under the key

This single mechanism is what makes "charge exactly once" achievable on top of an at-least-once world. State it first — it's the thing interviewers want to hear.

3. The ledger: source of truth

Money state is a double-entry ledger — an append-only record of entries, never updated in place. A payment is recorded as balanced entries (debit one account, credit another); a balance is the sum of entries. Append-only + double-entry gives:

• Auditability — the full history of every movement, reconstructible and tamper-evident.
• Correctness — balances are derived, not mutated, so you can't "lose" an update; every change is a new immutable entry.

You never overwrite a balance; you append an entry and recompute. This is the same event-sourced, immutable-log philosophy as the Stock Trading journal, applied to money.

4. Orchestrating the external gateway

The actual charge goes through a third-party gateway (Stripe/bank) — slow (seconds), external, and failable. Model each payment as a durable state machine / workflow (Multi-step Processes):

initiated → authorizing (call gateway) → authorized → captured → settled
          ↘ failed / timed-out → retry or mark failed (with idempotency)

Key rules:

• Persist intent before calling out. Write "payment initiated" durably before hitting the gateway, so a crash mid-call doesn't lose the fact that a charge may be in flight.
• Handle the ambiguous timeout. If the gateway call times out you don't know if it succeeded — never blindly retry; query the gateway's status (by your idempotency key) to find out, then record the true outcome. This ambiguity is the classic payment trap.
• Use a durable workflow engine (Temporal) or an explicit persisted state machine so an in-flight payment survives restarts and resumes exactly where it left off.

5. Reconciliation

Even with all this, your records and the gateway's/bank's can drift (a webhook missed, a timeout mis-recorded). Run a periodic reconciliation job: compare your ledger against the gateway's settlement reports and flag/repair discrepancies. In money systems, reconciliation is a first-class safety net, not an afterthought — it's how you prove correctness rather than assume it.

6. Deep dives

• Refunds/chargebacks: modeled as new ledger entries (reversing movements), never by deleting the original — history is immutable.
• Contention on a balance: concurrent operations on one account are serialized or use optimistic concurrency so a balance can't be corrupted by a race (Dealing with Contention).
• Webhooks: gateways confirm asynchronously via webhooks; process them idempotently (they can arrive late, twice, or out of order).
• PCI / secrets: never store raw card data; tokenize via the gateway. Secrets stay in a vault, never in code or the DB.

What to emphasise

Lead with idempotency, the ledger, and handling the gateway timeout:

"Correctness dominates, so I'd start with idempotency: every charge carries an idempotency key, and a repeat returns the original result instead of charging again — that's how you get exactly-once on top of retries and double-clicks. Money state is an append-only double-entry ledger, so balances are derived from immutable entries and everything is auditable; refunds are reversing entries, never deletions. The charge itself is a durable state machine around the external gateway: I persist intent before calling out, and on a timeout I don't blindly retry — I query the gateway by the idempotency key to learn the true outcome, since that ambiguity is where double-charges happen. A workflow engine keeps in-flight payments alive across restarts, and a reconciliation job continuously compares my ledger against the gateway to catch any drift."

Next up: Metrics Monitoring — ingesting and querying operational metrics.