Distributed Database: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
LOCAL_QUORUM read (~65% of traffic, hot data)
App reads a value. Coordinator selects 3 replicas, waits for 2 to respond (R=2). Fast path when all 3 replicas healthy.
App connects to any node; coordinator handles routing. Cluster-aware clients pick nearest node.
Hash partition key to token, look up replica placement via consistent hash + rack awareness. Prefer same-AZ replicas.
Same-AZ TCP hop to each of 3 selected replicas.
Cassandra memtable check → bloom filter → SSTable index → data. Compaction state affects tail.
Optimize: Bloom filter tuning: false-positive rate ~1% is standard. Lower FP rate = more RAM per SSTable.
Parallel with replica 1.
Third replica sometimes slower (compaction running, cache cold). LOCAL_QUORUM doesn't need to wait — take fastest 2.
LOCAL_QUORUM = R=2. Coord returns as soon as 2 responses match. Third response used for read repair (async).
Compare 2 responses (timestamp-based conflict resolution). Return newest. Trigger async read-repair if divergent.
Response delivery.
LOCAL_QUORUM read is **~10-40ms p99** because we wait for fastest 2 of 3 replicas. **The key architectural win**: we IGNORE the slowest replica. This is Dean & Barroso Tail-at-Scale applied at the database layer — same-shape solution as Elasticsearch hedged requests.
Bottleneck summary
Distributed database latency has THREE distinct profiles based on consistency level: **LOCAL_QUORUM (~10-40ms p99, most workloads)**, **hinted handoff writes (~10-30ms, resilient to single-node failures)**, and **EACH_QUORUM cross-DC (~100-300ms, price of multi-DC strong consistency)**. The architectural win of leaderless quorum: **we ignore the SLOWEST replica** (take fastest of N), which is Tail-at-Scale applied to databases. Compare to primary-based DBs where losing primary = writes fail.
Optimization tips (this architecture)
- **LOCAL_QUORUM is the default**: R=2, W=2 with RF=3 gives you consistency + availability + reasonable latency. Only escalate for critical data.
- **Tune bloom filters**: 1% FP rate standard. Lower FP = more RAM per SSTable. Higher FP = more disk reads on cache miss.
- **Commit log fsync policy**: `batch` with 1ms window is best balance. `periodic` faster but risks data loss on crash.
- **Hinted handoff**: Monitor hint queue. Long queues = cluster instability. Tune max_hints_delivery_threads.
- **Rack-aware replica placement**: Ensure replicas span racks/AZs. Same-AZ replication = fast but same-blast-radius.
- **Async cross-DC**: Use LOCAL_QUORUM + async cross-DC replication for 95% of workloads. EACH_QUORUM only for critical.
- **Compaction tuning**: Size-tiered vs leveled. Size-tiered = better write throughput. Leveled = better read latency + space efficiency.
- **Read repair**: Async repair when replicas diverge. Balances anti-entropy with latency.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.