Rate Limiter: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Allowed request — under limit (~95% of traffic)
User has budget remaining. Hierarchical Lua script checks per-user + per-endpoint + global, all under limit. Increments all three, returns 'allowed'.
Cluster-aware client hashes rate-limit key (hashtag ensures per-user + per-endpoint co-locate).
Persistent connection. Redis wire protocol. Sub-ms typical.
Optimize: Connection pooling avoids handshake. Same-AZ placement is critical.
Script: GET user_counter, GET endpoint_counter, GET global_counter, CHECK all < limits, INCR all three, SET TTL if new keys. Single atomic execution.
Optimize: EVALSHA is 10-20% faster than EVAL (script cached at Redis). Precompute SHA on client init.
Response with allowed=true + remaining count + reset timestamp.
Attach X-RateLimit-Remaining + X-RateLimit-Reset headers to response.
Allowed request check is **~1-3ms p99**. This is BELOW the sub-5ms budget. **The critical architectural pattern**: single Lua script does check+increment atomically for the entire hierarchy — this is why we co-locate per-user + per-endpoint keys on the same shard (hashtag routing). Cross-shard would require multiple round trips.
Bottleneck summary
Rate limiter latency is **DOMINATED BY REDIS ROUND-TRIP** (~1-3ms p99). Sub-5ms budget met with same-AZ Redis + EVALSHA + hierarchical Lua. **The critical architectural insight**: every API request calls the rate limiter, so this latency is on EVERY request. A 10ms rate limiter is a 10ms tax on the entire product. Sub-5ms is non-negotiable. **The fail-open policy** trades attacker exploitation for legitimate-user protection when Redis fails — this is the right call for 99% of use cases.
Optimization tips (this architecture)
- **Sub-5ms budget**: Every API request pays this cost. Latency is a first-class concern.
- **EVALSHA over EVAL**: Script cached at Redis. 10-20% faster.
- **Hierarchical Lua in single script**: Check per-user + per-endpoint + global atomically. Avoid multiple round-trips.
- **Hashtag routing**: Co-locate per-user + per-endpoint keys on same shard. Prevents cross-shard.
- **Aggressive timeout (3-5ms)**: Don't hang request path waiting for Redis. Circuit breaker after N failures.
- **Fail-open with degraded limit**: Over-serve rather than reject on infra failure. Consider degraded limit (10 RPS instead of 100) to prevent attacker exploitation.
- **Do NOT increment on rejection**: Rejected requests should not consume counter — unfair 'double penalty' next window.
- **Attach headers on all responses**: X-RateLimit-Remaining, X-RateLimit-Reset. Clients can implement backoff without polling.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.