Payment System: Junior → Architect Evolution
The mandated interactive flow. Step through each rung, ask what breaks FIRST, weigh the options, and defend the chosen architecture. This is how architectural thinking is learned — not by reading a fixed design, but by tracing how it evolves under growth pressure.
Scale Evolution Timeline
Step through 6 architectural rungs, from 10K RPS to 1B RPS. At each rung, ask: what will break FIRST? Why? What options exist? Which one do we pick — and what are we accepting?
This is the reasoning cycle that separates a Junior developer ("here's an architecture") from an Architect ("here's why this architecture, why now, and what breaks next").
10K RPS — single Postgres + ACID transactions
1. Current architecture
Where we are before growth pressure
3 × c6i.large apps + Postgres 15 db.r6i.large. Tables: `accounts(user_id, balance)`, `transactions(txn_id, from_account, to_account, amount, status, created_at)`. Every payment is a single Postgres transaction: BEGIN → SELECT balance FROM_ACC FOR UPDATE → UPDATE both accounts → INSERT into transactions → COMMIT. Idempotency via unique constraint on txn_id.
2. Growth trigger
What changed — the traffic/data force
MVP launch. 10K payments/sec peak. Payment volume: ~$100K/day. Simple wallet-to-wallet transfers, no external processor integrations yet.
3. Bottleneck — what breaks FIRST?
The component that saturates as growth arrives
None yet — Postgres ACID is exactly right
Payment integrity is the #1 requirement. Single-Postgres ACID transactions guarantee: (1) atomicity (money never disappears mid-transfer), (2) consistency (balance = sum of transactions), (3) isolation (two concurrent payments can't double-spend via SELECT FOR UPDATE), (4) durability (WAL sync on COMMIT). At 10K RPS this handles at ~30% CPU. p99 latency: 8-15ms.
Postgres CPU 30%, txn commit rate 8K/sec sustained, WAL sync latency 3ms, SELECT FOR UPDATE lock wait p99 = 2ms
4. Options — what could we do?
Alternatives an architect must consider before picking
- + Full ACID guarantees — no reasoning about eventual consistency
- + Zero new infrastructure
- + Single source of truth for money (no reconciliation needed)
- + Familiar tech stack (SQL, transactions)
- − Single point of failure — plan for 60s failover
- − Ceiling around 30-50K payments/sec on this hardware
- − Single-region — no APAC/EU regulatory support
- + Auditors love double-entry
- + Sets up for scale later
- − Premature complexity at 10K RPS
- − 2x database write cost
- − Team wastes cycles on ledger discipline that doesn't yet solve a problem
5. Chosen
The specific decision we're making
Do nothing — single Postgres ACID is genuinely enough at 10K payments/sec
6. Trade-offs
What we're explicitly accepting to move forward
- Accept single point of failure — plan for 60s failover during maintenance
- Accept eventual scale ceiling around 30-50K payments/sec (upgrade path is clear)
- Accept single region — no multi-region yet
- Accept no external processor integration (Stripe, Adyen, etc.) — direct wallet transfers only
7. New architecture
The system after this decision — headroom for the next 5-10x
3 apps → Postgres 15 with ACID transactions. Idempotency via unique constraint on txn_id. Total cost: $310/mo.
8. Next bottleneck — what will break at the NEXT rung?
This is the seed of the next rung
At ~100K payments/sec with external processor integrations (Stripe, PayPal, Adyen), synchronous transactions across services create cascading failure risk. Also auditors + regulators require double-entry bookkeeping for financial compliance. L5 shape.
What comes next in your Junior → Architect journey
You've traced 6 rungs of Payment System evolution. Now try the same reasoning cycle on a system you don't know yet — pick from the Systems catalog and answer the same questions: current arch → growth trigger → bottleneck → options → chosen → trade-offs → new arch → next bottleneck. That is architecture.