WhatsApp: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Same-region delivery — both users in US-East (~40% of messages)
Sender in New York, recipient in Los Angeles. Both connect to US-East Erlang cluster. Message delivered sub-100ms end-to-end.
User types message. App processes text, prepares payload.
Encrypt message with recipient's public key + current ratchet state. Server sees ciphertext only.
Optimize: Ratchet state cached on device. Fresh session establishment is slower (~50ms first message). Subsequent messages fast.
Cellular network latency. Persistent connection avoids handshake. Message payload is small (usually <1KB).
Optimize: TCP keep-alive at ~30s intervals. WiFi typically 10-30ms; cellular 100-200ms tail.
Erlang cluster gossip identifies recipient's home node. Same region — no cross-region routing needed.
Store ciphertext in Mnesia for recipient. Delivered when recipient online; queued if offline.
Same network characteristics as sender→server. Persistent connection means immediate push, not poll.
Decrypt with recipient's private key, verify signature, display message + update ratchet state.
Same-region delivery is **~150ms typical, ~500ms p99** — feels instant to users. The dominant latencies are CELLULAR NETWORK (both hops). E2E encryption + Mnesia + Erlang routing are <20ms combined. **WhatsApp's Erlang architecture achieves this on ~50 engineers** — proof that message-passing concurrency scales to billions.
Bottleneck summary
WhatsApp latency is **DOMINATED BY CELLULAR NETWORK** (both sender + recipient hops = 60-500ms). Cross-region adds ~150ms fiber transit — unavoidable physics. E2E encryption + Erlang routing + Mnesia persistence are <30ms combined. **Offline delivery has unbounded tail** because it depends on recipient's reconnection — this is why WhatsApp has 'sent' vs 'delivered' UI states.
Optimization tips (this architecture)
- **Persistent TCP connections**: XMPP-modified protocol keeps connections alive. Avoids handshake per message.
- **Regional Erlang clusters**: Sender/recipient reach nearest cluster fast; cross-region only crosses when both sides in different regions.
- **Signal Protocol Double Ratchet**: Session keys cached on device. Only first message per session pays setup cost.
- **Mnesia queue with 30-day TTL**: Guarantees eventual delivery. Purge stale messages to prevent unbounded growth.
- **APNS/FCM wake-up**: Silent push notification triggers app to reconnect. Improves offline-delivery latency by ~10x.
- **Delivery receipts**: Distinguish 'sent' (server received) from 'delivered' (recipient received) from 'read' (recipient opened). Sets correct user expectations.
- **Erlang gen_server concurrency**: One process per user. Cheap (~2KB per process). Enables millions of concurrent users per node.
- **Metadata-based spam**: Server sees WHO sends to whom (not content). Detect blast patterns without violating E2E.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.