Metrics Platform: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Metric scrape — Prometheus polling (100% of ingest)
Prometheus scrapes /metrics endpoint every 15s.
HTTP GET to /metrics.
Serialize + return.
Time-series append.
Remote write to M3DB.
Scrape is **~50-500ms end-to-end**. Async replication to long-term. 15s scrape interval balances freshness + cost.
Bottleneck summary
Metrics platform latency: **ingest ~50-500ms (scrape + append)**, **query ~100ms-2s (cardinality-bounded)**, **alert evaluation ~500ms**. Cardinality is #1 cost + latency factor. High-cardinality labels (user_id) cause explosion. **The critical trade**: retention length + resolution + cardinality bound cost + query performance.
Optimization tips (this architecture)
- **Cardinality limits per team**: Prevent explosion. 100K series max per team.
- **Precompute aggregates**: Common queries cached. 10x faster.
- **Time-range indices**: Only scan relevant time.
- **M3DB for high-cardinality**: Purpose-built. Scales to 100M+ series.
- **AI anomaly detection**: Replaces static thresholds. 60% fewer false-positives.
- **15s scrape interval**: Standard. Higher = missed spikes. Lower = 2x cost.
- **Histograms for percentiles**: t-digest, HDR-histogram. Accurate p99.
- **Long-term storage tiers**: 30d hot + 1y warm + 5y cold.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.