Distributed Database: Junior → Architect Evolution
The mandated interactive flow. Step through each rung, ask what breaks FIRST, weigh the options, and defend the chosen architecture. This is how architectural thinking is learned — not by reading a fixed design, but by tracing how it evolves under growth pressure.
Scale Evolution Timeline
Step through 6 architectural rungs, from 10K RPS to 1B RPS. At each rung, ask: what will break FIRST? Why? What options exist? Which one do we pick — and what are we accepting?
This is the reasoning cycle that separates a Junior developer ("here's an architecture") from an Architect ("here's why this architecture, why now, and what breaks next").
10K RPS — single Postgres (single-primary, no replicas)
1. Current architecture
Where we are before growth pressure
Postgres 15 on db.r6i.large (16 GB RAM, 500 GB gp3 SSD). ~10 tables. All reads + writes hit the primary. Automated daily snapshot + PITR (Point-In-Time Recovery) for last 7 days. Team writes normal SQL, joins whatever they want.
2. Growth trigger
What changed — the traffic/data force
Series A startup, ~50K DAU. 10K RPS peak (mostly during business hours). Read:write ratio ~10:1. Data set ~50 GB. Query pattern is well-served by proper indexing.
3. Bottleneck — what breaks FIRST?
The component that saturates as growth arrives
None yet — Postgres single-primary is comfortable
db.r6i.large handles 10K RPS at ~25% CPU when queries are well-indexed. 50 GB fits in buffer cache. Single-connection latency: 0.5-3ms for indexed lookups; 5-30ms for well-planned joins. Team can use ACID transactions, foreign keys, complex joins — full relational power. This is the RIGHT architecture for this scale.
Postgres CPU 25%, buffer cache hit rate 98%, connection count peak 80/300, WAL write rate 2 MB/s, p99 query latency 12ms
4. Options — what could we do?
Alternatives an architect must consider before picking
- + Zero new infrastructure
- + Full ACID + joins + foreign keys + arbitrary queries
- + Team already knows SQL — no new mental model
- + Postgres is battle-tested at this scale (millions of production deployments)
- − Single point of failure — plan for 60s failover during maintenance
- − Ceiling near 50-100K RPS on this hardware
- − No geographic distribution — one region only
- + Reads scale across replicas
- + HA on primary failure
- − Premature complexity — replicas add lag (50-500ms), stale reads
- − 3x DB cost for headroom we don't need yet
5. Chosen
The specific decision we're making
Do nothing — single Postgres primary is genuinely the right architecture for 10K RPS
6. Trade-offs
What we're explicitly accepting to move forward
- Accept single point of failure — 60s downtime on primary failover is acceptable at this scale
- Accept 1 region only — no APAC/EU users yet, no geographic distribution needed
- Accept 50-100K RPS ceiling — plan for the migration when we hit ~50K RPS
7. New architecture
The system after this decision — headroom for the next 5-10x
10 app instances → PgBouncer connection pooler → Postgres 15 on db.r6i.large + gp3 SSD + daily snapshots + PITR. Cost: $310/mo.
8. Next bottleneck — what will break at the NEXT rung?
This is the seed of the next rung
At ~100K RPS (Series B growth), read load saturates single-primary CPU. Also 50 GB data set grows to 500 GB — no longer fits in RAM buffer cache. Read replicas + tuned indexes — L5 shape.
What comes next in your Junior → Architect journey
You've traced 6 rungs of Distributed Database evolution. Now try the same reasoning cycle on a system you don't know yet — pick from the Systems catalog and answer the same questions: current arch → growth trigger → bottleneck → options → chosen → trade-offs → new arch → next bottleneck. That is architecture.