Distributed Cache: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Cache hit — Redis GET (~90% of reads)
App requests a cached value. Cluster-aware client routes to correct shard. Redis serves from in-memory hash → sub-millisecond.
Cluster-aware client hashes key to determine target shard. Pure CPU work, no network. Nanosecond-scale.
Cross-instance TCP hop within same AZ. Connection pool reused (no handshake). Persistent HTTP/2 or Redis wire protocol.
Optimize: Same-AZ placement (Redis in same AZ as app) is critical. Cross-AZ adds 0.5-1ms of noise. Use connection pooling to avoid TCP handshake.
Redis single-threaded event loop dequeues command, hashes to slot, retrieves from local memory. Tail comes from occasional RDB snapshot or AOF fsync.
Optimize: Set save '' (disable RDB) + appendonly no (or AOF everysec, not always) for cache workloads. Snapshot pauses cause p99.9 spikes.
Response delivery over same connection.
Bytes → object graph. JSON parse for small values, MessagePack/Protobuf for larger/hot values. Tail from occasional GC pause during parse.
Optimize: Prefer binary formats (MessagePack, Protobuf, Cap'n Proto) for hot paths. JSON parsing is 3-10x slower for equivalent data.
Cache hit is ~1ms end-to-end at p50. Even the p99 stays under 5ms because Redis is genuinely fast and network is same-AZ. **Note the p999 jump** — that's Redis snapshots, GC, and rare TCP retransmits. If your cache p99.9 is way higher than p99, look at RDB/AOF settings FIRST.
Bottleneck summary
Distributed cache latency has bimodal distribution: **cache hit is ~1ms** (dominated by same-AZ TCP + Redis event loop), **cache miss is ~15-30ms** (dominated by Postgres query + PgBouncer queue). Since 90%+ of traffic is cache hit, aggregate p99 tracks cache-hit p99. **Improving hit rate is 10x more valuable than optimizing cache-hit latency.** The tail (p99.9) is dominated by Redis RDB snapshots, GC pauses, and PgBouncer queue depth — all fixable with tuning.
Optimization tips (this architecture)
- **Same-AZ placement**: Redis in the same AZ as app cuts network noise 3-5x. Cross-AZ adds 0.5-1ms of jitter per hop.
- **Disable RDB snapshots**: For pure cache workloads, set save '' + appendonly no. Snapshots cause 100-500ms pause during BGSAVE.
- **Connection pooling everywhere**: Redis clients keep persistent connections; PgBouncer transaction-mode pool for Postgres. Avoid handshake latency.
- **Fire-and-forget cache writes**: Never block response on SETEX. Async cache writes decouple write path from cache hydration.
- **Cluster-aware clients**: Use jedis/ioredis with cluster support. Client-side sharding eliminates proxy hops.
- **Buffer cache tuning**: shared_buffers = 25% RAM, effective_cache_size = total RAM. Warm cache after restart with pg_prewarm.
- **Binary serialization**: MessagePack/Protobuf 3-10x faster than JSON. Use for hot paths, keep JSON for cold admin paths.
- **Improve hit rate**: 90% → 99% hit rate cuts p99 tail more than 10x. Look at cache size, TTL, warm-up strategy.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.