Distributed Cache: Junior → Architect Evolution
The mandated interactive flow. Step through each rung, ask what breaks FIRST, weigh the options, and defend the chosen architecture. This is how architectural thinking is learned — not by reading a fixed design, but by tracing how it evolves under growth pressure.
Scale Evolution Timeline
Step through 6 architectural rungs, from 10K RPS to 1B RPS. At each rung, ask: what will break FIRST? Why? What options exist? Which one do we pick — and what are we accepting?
This is the reasoning cycle that separates a Junior developer ("here's an architecture") from an Architect ("here's why this architecture, why now, and what breaks next").
10K RPS — local in-app LRU (no external cache)
1. Current architecture
Where we are before growth pressure
3 × app instances (c6i.large) behind ALB → Postgres RDS db.r6i.large. Each app has an in-process Guava/Caffeine LRU cache with 10K entries, 60s TTL. Cache-aside pattern: check local cache → if miss, query DB → populate cache → return.
2. Growth trigger
What changed — the traffic/data force
Early product launch. 10K RPS sustained. Read:write ratio is 100:1 (typical B2C read-heavy workload). Postgres CPU sits at 25% with the 90%+ local-cache hit rate absorbing the read load.
3. Bottleneck — what breaks FIRST?
The component that saturates as growth arrives
None yet — local cache is doing its job
At 10K RPS with 90%+ cache hit rate on hot items (Zipfian distribution), Postgres sees only ~1K RPS — well under its 30K QPS ceiling. Each app instance holds its own 10K-entry cache in ~50 MB of process memory. p99 latency: 3-8ms (local cache hit) or 12-25ms (cache miss).
app process RSS 200 MB (50 MB cache), local cache hit rate 92%, DB CPU 25%, p99 latency 8ms
4. Options — what could we do?
Alternatives an architect must consider before picking
- + Zero infrastructure to add
- + Zero cache-invalidation complexity — local cache is per-process, no cross-instance coordination
- + Fastest possible cache lookup: in-process memory (nanoseconds)
- − Cache is per-instance — 3 apps mean 3 independent caches, some duplication of hot items
- − No cross-instance visibility — a write on app-1 doesn't invalidate app-2's cache (60s TTL is the safety net)
- − On restart, cache is cold — 30-60s of degraded latency until it warms
5. Chosen
The specific decision we're making
Do nothing — local Guava/Caffeine LRU is genuinely enough at 10K RPS
6. Trade-offs
What we're explicitly accepting to move forward
- Accept 60s cache-invalidation staleness on writes (TTL-based, not event-based)
- Accept per-instance cache duplication (~3x hot-item memory usage)
- Accept 30-60s cold-start penalty on app restart
- Accept no shared cache — cannot use for cross-request state (session data still hits DB)
7. New architecture
The system after this decision — headroom for the next 5-10x
3 × c6i.large apps (each with Caffeine LRU) → ALB → db.r6i.large Postgres. Total cost: $580/mo (apps $300 + DB $280). Boring, cheap, works.
8. Next bottleneck — what will break at the NEXT rung?
This is the seed of the next rung
At ~100K RPS with data set growing past 500K distinct items, local LRU eviction thrashes — hit rate drops from 92% → 60%. DB sees 40K RPS, saturates. Need a SHARED cache tier — the L5 problem.
What comes next in your Junior → Architect journey
You've traced 6 rungs of Distributed Cache evolution. Now try the same reasoning cycle on a system you don't know yet — pick from the Systems catalog and answer the same questions: current arch → growth trigger → bottleneck → options → chosen → trade-offs → new arch → next bottleneck. That is architecture.