Building a Payment Gateway from Scratch: What the Tutorials Don't Tell You

Building a payment gateway simulation taught me more about distributed systems than any course I've taken. Not because payments are uniquely complex — but because the consequences of getting it wrong are immediately obvious and unambiguous. Money is either moved or it isn't.

Idempotency is not optional

The first principle of payment engineering is idempotency. Networks are unreliable. Clients retry. If your POST /authorize endpoint processes the same request twice, you've double-authorized — which in a real system means double-charging a customer.

The fix is idempotency keys: a unique identifier the client generates per logical transaction, sent as a header. The server stores this key alongside the result. On a retry with the same key, return the stored result without processing again.

// Pseudocode — the actual implementation uses PostgreSQL
const existing = await db.transactions.findByIdempotencyKey(key);
if (existing) return existing.response;

const result = await processAuthorization(payload);
await db.transactions.save({ key, response: result });
return result;

The tricky part: storing the idempotency key and the result must be atomic. If your process crashes between processing and saving, the next retry should re-process, not return a half-baked response.

Webhook delivery is harder than it looks

When a payment completes, you need to notify downstream systems (inventory, fulfillment, accounting). The naive approach — fire an HTTP request from your completion handler — fails constantly. The downstream system might be down, slow, or returning errors.

The production pattern is a delivery engine with exponential backoff and a dead-letter queue:

1. On payment completion, write a webhook event to a delivery queue
2. A background worker attempts delivery with retry delays: 10s, 30s, 2min, 10min, 1hr
3. After N failures, move to the dead-letter queue for human review
4. On successful delivery, the downstream system acknowledges with a 2xx and the worker marks it done

The downstream system must be idempotent too — it will receive the same webhook multiple times during retries.

Row-level locking prevents race conditions

Consider two concurrent requests trying to capture the same authorized payment. Without locking, both read the AUTHORIZED status, both decide to proceed, and you end up with a double capture.

PostgreSQL's SELECT ... FOR UPDATE acquires a row-level lock:

BEGIN;
SELECT * FROM payments WHERE id = $1 FOR UPDATE;
-- Now only this transaction can modify this row until COMMIT
UPDATE payments SET status = 'CAPTURED' WHERE id = $1;
COMMIT;

The second concurrent request blocks until the first commits, then reads the CAPTURED status and correctly rejects the duplicate.

The state machine you need to get right

A payment follows a strict state machine:

• PENDING → AUTHORIZED (on successful authorization)
• AUTHORIZED → CAPTURED (charge the card)
• AUTHORIZED → VOIDED (cancel before capture)
• CAPTURED → REFUNDED (return funds)

Any transition that violates this machine — capturing a voided payment, voiding a captured one — should be a hard error at the API layer, not a database constraint failure. Validate state transitions explicitly in your service layer.

What I'd do differently

I'd add an event log from day one. Instead of a single status column that overwrites itself, every state transition gets its own immutable row with a timestamp and metadata. The current state is just the most recent event. This makes debugging far easier and enables replay-based recovery.