Netflix: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Popular title — OCA hit (~95% of streams)
User picks a popular title. Manifest cached at OCA inside their ISP. Video segments prefetched. Sub-500ms to first frame.
Metadata request (title info, user watchlist, resume position). Catalog stored in AWS regions — this is NOT served from OCA. Cross-region latency for non-US users.
Optimize: Prefetch metadata during app launch. Cache watchlist locally in app. This hop happens once per session, not per stream.
Netflix control plane routes client to nearest OCA using ISP + geo signals. Returns OCA IP + auth token.
Optimize: Client caches OCA IP for 5-15 min. First video load pays this cost; subsequent titles in same session skip it.
OCA inside ISP network — typically <20ms RTT to subscriber. Manifest is small (~2KB). Cache hit at OCA is near-guaranteed for popular titles.
Optimize: OCAs pre-populate hot content nightly. Rare titles may require OCA-to-region fill (slower first fetch).
Player parses HLS manifest, checks buffer, selects bitrate based on measured bandwidth. Complex client-side logic.
First segment ~1MB. OCA serves from local SSD/RAM. Bandwidth-limited (need 8Mbps for HD). Segment TTFB <50ms typical.
Optimize: Pre-fetch next segment before current finishes (~1s ahead). Use CMAF for chunked encoding — start playback with first 2s of segment.
Hardware-accelerated decode. First frame shown to user. This is the 'time to first frame' user actually sees.
Optimize: H.264 decode on all devices. HEVC/AV1 on newer devices — same bitrate, better quality but slower initial decode.
OCA-hit path is **sub-200ms to first byte + ~500ms to first frame**. The dominant cost is CLIENT-SIDE (decode + render + ABR logic), NOT infrastructure. **This is why 'the last mile' matters more than 'the cloud'** — Netflix optimizes hardware decoders, not just servers.
Bottleneck summary
Netflix time-to-first-frame is DOMINATED BY CLIENT-SIDE DECODE + RENDER (~500ms). Infrastructure (catalog fetch + OCA delivery + segment fetch) contributes ~200ms combined for OCA hits. **The critical architectural win** is putting caches INSIDE ISP networks (OCAs) — this eliminates cross-region hops for 95%+ of traffic. Long-tail titles pay the OCA-fill penalty once but subsequent viewers benefit.
Optimization tips (this architecture)
- **OCA density**: Netflix runs 15,000+ OCAs inside ISP networks globally. More OCAs = closer to users. This is the #1 latency win.
- **Nightly pre-fill**: Predict tomorrow's popular content (ML on watch history) and pre-fill OCAs. Reduces long-tail fills.
- **CMAF chunked encoding**: Start playback with first chunk (100ms) instead of waiting for full segment (2s). Massive TTFF win.
- **Client-side ABR**: Hysteresis on bandwidth measurement. Don't flip-flop between bitrates. Prefer slight quality drop over frequent switches.
- **Hardware decode**: HEVC/AV1 decode 5-10x faster in hardware than software. Use HEVC on iPhone/Roku, H.264 for compatibility fallback.
- **Pre-fetch metadata**: Load catalog + user data during app launch, not on stream start. Amortize cost.
- **Buffer strategy**: 30-60s buffer typical. Longer = smoother playback under network variance, shorter = faster seek.
- **Multi-bitrate**: Encode 480p/720p/1080p/4K + HDR variants. Storage cost <10x video size but massive UX win for network variability.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.