Back to Rate Limiter
MASTERCLASS
Gold-standard deep diveRate Limiter — Masterclass
Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.
1. Architecture Decision Record
The format working architects use to document a decision so future teams understand context, options, and reversal conditions.
ADR 001
Architecture Decision RecordRedis Cluster (sharded) + hierarchical Lua atomic script (via EVALSHA) + hashtag routing for co-located hierarchies + fail-open with degraded limit fallback — over EVAL (uncached), naive-per-instance token bucket, CRDT-only counters, sticky-routing per user, or third-party (Cloudflare Rate Limiting).
Context
We are building a rate-limiting service for a multi-tenant API platform. Every API request passes through the rate limiter — sub-5ms budget is CRITICAL. Peak: 1M RPS across 100K+ unique API keys. Hierarchical limits: per-user AND per-endpoint AND global must all check atomically. Multi-tenant: some users burst 100 RPS, others sustained 1000 RPS. Reference deployments: Stripe (~250K RPS with sub-5ms rate limit), Cloudflare (~100M RPS at edge PoPs with per-PoP counters), Google APIs (billions of RPS with adaptive-limit fallback). Failure model: Redis SPOF creates cascading unavailability; the fail-open vs fail-closed decision is architectural + political.
Constraints
- 1M RPS baseline, 5M+ RPS peak (during customer bursts)
- 100K+ unique API keys across ~500 tenants
- Hierarchical limits: per-user + per-endpoint + global (3 checks per request)
- p99 check latency: <5ms (every API request pays this cost)
- Fail-open policy: over-serving > rejecting legitimate users on our infra failure
- Accuracy: 5% over-serve during merge windows is acceptable; blatant over-serve (10x limit) is not
- Support burst allowance: users can spike 5x limit for 10 seconds
- Multi-tenant isolation: one tenant hitting limits should not affect other tenants' latency
Options considered
Redis Cluster + hierarchical Lua atomic script (EVALSHA cached) (chosen)
Pros
- Single Lua script atomically checks per-user + per-endpoint + global in ONE round-trip
- EVALSHA is 10-20% faster than EVAL (script cached at Redis)
- Same-AZ TCP + Redis event loop = sub-2ms p99
- Hashtag routing co-locates per-user + per-endpoint keys on same shard (no cross-shard)
- Redis Cluster scales horizontally to 16M+ ops/sec
- Fail-open with degraded fallback (10 RPS instead of 100) prevents attacker exploitation
- Stripe + Cloudflare + Google actual pattern
Cons
- Redis SPOF per shard — plan for shard failover
- Lua script authoring is harder to test than SDK code
- Hashtag routing convention required across all clients
- Cross-shard queries (analytics: 'top rate-limited tenants') require Redis SCAN + client aggregation
Cost: $3.5K/mo (16-shard Redis Cluster + Sentinel for HA + monitoring)
Redis + EVAL (uncached Lua)
Pros
- Simpler client code (no script hash management)
- Same atomic semantics as EVALSHA
Cons
- Redis parses Lua on every EVAL call — 10-20% slower
- At 1M RPS baseline, that 10-20% = $10-20K/mo extra Redis capacity
- Same architecture otherwise — just slower
Cost: $4K/mo (same infrastructure but higher CPU)
Naive per-instance token bucket (Guava RateLimiter / golang.org/x/time/rate)
Pros
- Sub-microsecond check latency (in-process, no network)
- Zero external infrastructure
- Team already knows the libraries
Cons
- Over-serves proportional to instance count (30 instances = 30x limit)
- Cannot enforce global rate limits
- Bucket state lost on restart (fresh window every deploy)
- Cannot support hierarchical limits without shared state
Cost: $0 (library code) but architecturally broken above ~50K RPS
CRDT-only counters (G-Counter with async merge)
Pros
- No SPOF — fully distributed
- Fault-tolerant to any single-node failure
- Regional locality — no cross-region synchronous coordination
Cons
- 5-10% over-serve during merge windows (10-60 seconds)
- Complex reasoning about eventual consistency at check time
- Only makes sense at multi-region scale (L7+); overengineering at L6
- Doesn't help with sub-5ms latency budget (still need Redis for check)
Cost: $25K/mo (multi-region Redis + CRDT merger workers)
Sticky routing (same user → same instance)
Pros
- Preserves in-process bucket accuracy
- No Redis dependency
Cons
- Load balancer complexity (hash-by-user routing)
- Instance hotspot: popular users overload specific instances
- Fails on instance restart (user's state lost mid-window)
- Doesn't work for hierarchical limits (per-endpoint routing conflicts with per-user)
Cost: $0 marginal but architecturally brittle
Third-party (Cloudflare Rate Limiting)
Pros
- Zero infrastructure to run
- 300+ edge PoPs — global sub-30ms
- DDoS protection built-in
- Cloudflare team handles all ops
Cons
- $0.05 per million requests × 1M RPS × 86400s × 30d = $130K/mo at sustained baseline
- Vendor lock-in on foundational infrastructure
- Custom logic (per-endpoint burst allowances, tenant reputation) hard to express
- Latency depends on Cloudflare's edge — typically <30ms globally but not <5ms
Cost: $130K+/mo at 1M RPS sustained — 30x more expensive than self-hosted
Chosen solution
Redis Cluster + hierarchical Lua atomic script + EVALSHA + hashtag routing + fail-open with degraded limit fallback
Why
Sub-5ms latency budget is DECISIVE. Only in-process buckets or same-AZ Redis meet it. In-process buckets fail architecturally above ~50K RPS (nx over-serve). Cloudflare's edge PoPs are ~30ms — 6x over budget. So the choice is Redis + Lua. Within Redis: hierarchical Lua atomic (single Lua script checks per-user + per-endpoint + global in one round-trip) is the only way to enforce hierarchy without multiple round-trips that blow the budget. Hashtag routing ({user_id}:endpoint keys) co-locates hierarchy keys on same shard — avoids cross-shard atomic ops (impossible in Redis Cluster). EVALSHA caches the script at Redis — 10-20% faster than EVAL, which at 1M RPS is $10-20K/mo saved. Fail-open with degraded limit (10 RPS instead of 100 when Redis unreachable) balances 'don't reject legitimate users on our infra failure' with 'don't let attackers exploit fail-open with unlimited requests'.
Rejected alternatives (with reasons)
- Cloudflare Rate Limiting — $130K+/mo at our scale (30x self-hosted); vendor lock-in on foundational infra
- Naive per-instance token bucket — architecturally broken above ~50K RPS (n-instance over-serve)
- CRDT-only counters — overengineering at single-region L6; 5-10% over-serve during merge window
- Sticky routing — brittle (instance hotspots, restart data loss); doesn't work for hierarchical
- EVAL (uncached Lua) — 10-20% slower = $10-20K/mo extra Redis capacity vs EVALSHA
Trade-offs accepted
- Accept Redis SPOF per shard — plan for shard failover (30-60s of degraded-limit fallback)
- Accept Lua script authoring discipline (harder to test than SDK code)
- Accept hashtag routing convention across all clients (documented, enforced by client library)
- Accept cross-shard analytics queries require SCAN + client aggregation (rare, batched)
- Accept 3ms Redis timeout as aggressive but non-negotiable (falls back to degraded limit on timeout)
- Accept fail-open policy over fail-closed — attacker exploitation mitigated by degraded limit
- Accept dedicated Rate Limiting Platform team (5-10 engineers) — foundational infra warrants first-class ownership
Consequences
- Every API request pays 1-3ms rate-limit latency — this is the tax on the entire product
- Redis Cluster becomes critical-path infrastructure — 24/7 SRE + monitoring
- Hashtag routing convention flows through every client library + service integration
- Fail-open policy documented and understood by security + compliance teams
- Redis shard failover triggers degraded-limit fallback for ~30-60s — customer-visible during incidents
- Attackers know we fail-open — deliberate DDoS + Redis-DOS is a real threat vector (mitigated by degraded limit)
- Rate-limiting logic becomes reusable primitive for feature flags, tenant quotas, spam control
When would we reverse this decision?
- If Redis Cluster ops cost exceeds 3 engineers dedicated → evaluate Cloudflare enterprise (accept 30x cost)
- If we expand to <100K RPS steady after cost cutting → naive per-instance may be acceptable again (regressive but simpler)
- If multi-region expansion + sub-30ms cross-region rate limits required → CRDT counters unavoidable
- If we get acquired by Cloudflare → their infrastructure becomes free
2. Business constraint exercise
Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.
Business constraint exercise
You are the platform engineering lead at a $50M ARR B2B SaaS company (200 engineers). CTO drops in Monday: 'Cloudflare Rate Limiting proposal: $200K/year, 300+ edge PoPs, DDoS included, zero infrastructure. Alternative: build our own on Redis. Which is right? Board decision Wednesday.'
Constraints
- 1$50M ARR B2B SaaS (150 enterprise customers)
- 2API is 40% of revenue — rate limiting failure = major incident
- 3500K RPS baseline, 5M RPS during customer batch runs
- 45-engineer Platform team (of 200 total engineers)
- 5Existing infrastructure: AWS us-east-1 primary, EU-west failover
- 63-year budget horizon (board wants forecast)
- 7Existing customer: 3 largest customers do batch runs at 3 AM (each 500K RPS burst for 2 hours)
- 8Cloudflare proposal: $200K/yr = $0.05 per million requests + $50K/yr enterprise support
- 9DDoS protection: currently paying $30K/yr AWS Shield Advanced (marginal but existing)
Your question
What do you recommend to the board Wednesday? Show 3-year cost projection + operational cost + strategic trade-offs. Anticipate the board's follow-up: 'why not both?' and address it.
3. Production incident scenario
You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.
INCIDENT
Redis Cluster shard failure during Black Friday flash-sale — fail-open policy exploited by attackersPagerDuty alert at 12:47pm ET on Black Friday. Redis Cluster shard-7 (of 16) failed hard (kernel panic). Failover to replica in progress. Meanwhile: Rate Limiter fail-open policy triggered — 6.25% of API traffic bypassing limits for shard-7 keys. Marketing dashboards show anomalous traffic spike from suspicious IPs. Attackers may be exploiting the fail-open window.
Metrics
- redis.shard7.status: DOWN at 12:47:03 (kernel panic on primary)
- redis.shard7.failover_status: in_progress (target: 30-60s)
- rate_limiter.failure_count.per_second: 62K (6.25 pct of 1M RPS shard-7 traffic)
- rate_limiter.fail_open_allowed_percent: 6.25 pct (shard-7 keys only)
- api.total_requests_per_second: 1.15M (was 1M baseline — 15 pct spike above expected)
- api.top_endpoint_requests.suspicious: /v1/coupons/apply spike 40x baseline (12K/sec, was 300/sec)
- api.tenant_id.7f3e2a: 480K RPS (was 2K RPS baseline — 240x spike from single tenant)
- iam.suspicious_ip_range: 47.201.0.0/16 hitting 45K RPS from 12:47 onwards (was 200 RPS baseline)
- downstream.checkout_service.cpu_percent: 92 (was 40 baseline — attacker traffic reaching origin)
- downstream.database.connection_pool_wait_p99: 800ms (was 20ms baseline)
Logs
- 12:47:03 redis-shard7-primary: 'kernel panic - not syncing: Fatal exception'
- 12:47:04 rate-limiter-service: 'shard-7 timeout after 3ms — triggering fail-open policy'
- 12:47:05 rate-limiter-service: 'ALERT: fail-open triggered for shard-7 keys (6.25 pct of traffic)'
- 12:47:12 iam-service: 'anomalous 100x traffic spike from IP range 47.201.0.0/16'
- 12:47:15 checkout-service: 'connection pool at 95 pct — queueing requests'
- 12:47:23 rate-limiter-service: 'shard-7 failover to replica in progress (ETA 30s)'
- 12:47:45 marketing-dashboard: 'anomalous coupon-apply traffic 40x baseline'
- 12:47:58 redis-shard7-replica: 'promoted to primary'
- 12:48:03 rate-limiter-service: 'shard-7 recovered — fail-open policy disengaged'
Dependency health
- Redis shard-7: DOWN → RECOVERING (failover in progress, 30-60s)
- Redis shards 0-6, 8-15: HEALTHY — 93.75 pct of traffic unaffected
- Rate limiter service: DEGRADED — fail-open for 6.25 pct of traffic
- API gateway: HEALTHY at LB layer but seeing 1.15M RPS (vs 1M baseline)
- Checkout service: DEGRADED — 92 pct CPU (was 40 pct), connection pool 95 pct
- Payment service: HEALTHY but seeing suspicious /v1/coupons/apply traffic
- Postgres: DEGRADED — connection pool wait p99 800ms
- Fraud detection: HEALTHY — flagging 47.201.0.0/16 as suspicious
Your investigation
1
It's 12:47:15pm. Fail-open policy is triggered for shard-7 keys — 6.25% of traffic. Marketing dashboards show a 40x spike on /v1/coupons/apply. What's your SINGLE highest-priority action in the next 60 seconds?
Hint: The fail-open policy is doing what it was designed for. But 40x on coupons is suspicious. What can you change WITHOUT waiting for Redis recovery?
2
Assuming degraded fallback is now active. IP range 47.201.0.0/16 is doing 45K RPS from previously-idle IPs. What's your next action?
Hint: This is either coordinated attack traffic OR a customer legitimately using a cloud provider CDN. How do you distinguish + respond?
3
Redis shard-7 recovered at 12:48:03. Fail-open policy disengaged. But between 12:47:03 and 12:48:03, some attackers got through the fail-open + degraded-limit window. How do you audit the damage?
Hint: Fail-open logs + downstream service logs. Cross-reference.
4
Postmortem the next day. Name 3 action items ranked by impact, with quantified reduction in incident probability or blast radius for each.
Hint: Some are technical, some are process. Prioritize by risk-adjusted impact.
5
Draft the customer-facing status page update at T+10 minutes (12:57pm — after Redis recovery + attack mitigation). Constraint: honest, no jargon, no over-promising, no 'never again' language.
Hint: Users saw brief slowness and possibly saw coupon-abuse. Set correct expectations.
Knowledge graph
Learn these first
- Redis atomic operations + Lua scripting (EVALSHA vs EVAL)
- Token bucket vs sliding window vs fixed window algorithms
- Hierarchical rate limits (per-user AND per-endpoint AND global)
- Fail-open vs fail-closed on infrastructure failure
Where this appears in the curriculum
This is the Gold Standard.
Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.