Rate Limiter: Junior → Architect Evolution
The mandated interactive flow. Step through each rung, ask what breaks FIRST, weigh the options, and defend the chosen architecture. This is how architectural thinking is learned — not by reading a fixed design, but by tracing how it evolves under growth pressure.
Scale Evolution Timeline
Step through 6 architectural rungs, from 10K RPS to 1B RPS. At each rung, ask: what will break FIRST? Why? What options exist? Which one do we pick — and what are we accepting?
This is the reasoning cycle that separates a Junior developer ("here's an architecture") from an Architect ("here's why this architecture, why now, and what breaks next").
10K RPS — in-process token bucket (per-instance)
1. Current architecture
Where we are before growth pressure
3 × c6i.large app instances. Each instance holds an in-memory token bucket per API key using Guava RateLimiter or Golang golang.org/x/time/rate. No shared state — each instance enforces limits independently.
2. Growth trigger
What changed — the traffic/data force
Early API product. 10K RPS total, 100 API keys, ~100 requests/key/second average. Peak per-key: 500 RPS during customer bursts.
3. Bottleneck — what breaks FIRST?
The component that saturates as growth arrives
None yet — in-process bucket is fine
Per-instance limits: user with 100 RPS budget can burst to 300 RPS if hitting 3 instances simultaneously (100 per instance). Acceptable at this scale — 3x over-serve is fine for early product. In-memory bucket = sub-microsecond check, no external calls.
check latency <1μs (in-process), 3x over-serve possible under load balancer split, memory 10 MB per instance for 100 keys
4. Options — what could we do?
Alternatives an architect must consider before picking
- + Sub-microsecond check latency (in-process, no network)
- + Zero new infrastructure — pure library code
- + Handles 10K RPS trivially at any instance count
- + Team ships fast — Guava/x-time-rate is one import
- − Nx over-serve where N = instance count (users hit ALL instances)
- − Cannot enforce global limits (only per-instance)
- − Bucket state lost on restart (fresh window every deploy)
- − No visibility into limit hits across the fleet
5. Chosen
The specific decision we're making
Do nothing — in-process token bucket is genuinely enough at 10K RPS
6. Trade-offs
What we're explicitly accepting to move forward
- Accept Nx over-serve where N = instance count (3x with 3 instances)
- Accept per-instance state (bucket resets on restart)
- Accept no cross-instance coordination
- Accept ceiling around 100K RPS + few dozen instances before over-serve is intolerable
7. New architecture
The system after this decision — headroom for the next 5-10x
3 × c6i.large apps with Guava RateLimiter (JVM) or golang.org/x/time/rate. Per-instance token bucket per API key. Total cost: $0 marginal (library code).
8. Next bottleneck — what will break at the NEXT rung?
This is the seed of the next rung
At ~100K RPS with 30+ instances, per-instance limits become useless: user gets 30x their intended limit by fanning out across instances. Need shared state — Redis. L5 shape.
What comes next in your Junior → Architect journey
You've traced 6 rungs of Rate Limiter evolution. Now try the same reasoning cycle on a system you don't know yet — pick from the Systems catalog and answer the same questions: current arch → growth trigger → bottleneck → options → chosen → trade-offs → new arch → next bottleneck. That is architecture.