Slack: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Message send — same channel (~85% of traffic)
User sends message in a channel. Persisted to Vitess, fanned out via Redis Pub/Sub to connected recipients. Sub-100ms end-to-end.
Persistent WebSocket connection — no handshake. Message sent as WebSocket frame.
Optimize: Persistent connections eliminate TCP+TLS handshake (~100ms savings). WebSocket ping/pong every 30s keeps connection alive through NAT.
WebSocket gateway looks up workspace → shard mapping (cached).
Write message row to workspace's Vitess shard. MySQL primary + async replica.
Optimize: Use single-shard transactions when possible. Cross-shard operations are 10x slower via 2PC.
PUBLISH to channel-specific Redis Pub/Sub topic. All gateway processes subscribed to that channel receive.
Optimize: Redis Pub/Sub is at-most-once — if subscriber disconnects, message lost. Slack has to reconcile via Vitess replay on reconnect.
Sender's client shows 'sent' checkmark.
All gateways with connected recipients in this channel receive Pub/Sub message.
Push message to each connected client. Parallel across recipients.
Slack message delivery is **~50-200ms p99** — persistent WebSocket eliminates handshake. **The two writes (Vitess + Pub/Sub) are parallel from the gateway's perspective** — sender's ack fires as soon as either completes. This is the pattern for 'feels instant' messaging: minimize round-trips + use persistent connections + fanout via pub/sub.
Bottleneck summary
Slack messaging is **DOMINATED BY PERSISTENT WEBSOCKET LATENCY** (~15-100ms per hop). Message send is sub-100ms because Vitess write + Redis Pub/Sub happen in parallel. Typing indicators skip Vitess entirely (Pub/Sub only). **The critical failure mode**: reconnect storms during network blips or deploys. Slack's Ringpop + backoff + jitter architecture handles this — without it, a 30-second network blip = cluster-wide overload.
Optimization tips (this architecture)
- **Persistent WebSocket connections**: Eliminate TCP+TLS handshake per message (~100ms savings). Ping/pong keeps NAT alive.
- **Parallel Vitess + Pub/Sub writes**: Sender ack fires on first write. Both writes complete in parallel.
- **Sharded by workspace**: Vitess vindex + WebSocket gateway sharding. Same workspace = same shard = no cross-shard queries.
- **Ephemeral events via Pub/Sub only**: Typing indicators, presence updates. Never write to DB. 10x throughput vs persistent writes.
- **Cap backfill at 1000 messages**: Older = summary + scroll. Prevents pathological queries after long offline periods.
- **TLS session resumption**: Reconnect is 10x more expensive than message send. Reuse TLS session where possible.
- **Ringpop for gateway routing**: Consistent hashing across gateways. Handle gateway failures without dropping all connections.
- **Jittered reconnect backoff**: Prevents reconnect storms after network blips or deploys.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.