Search Autocomplete: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Cached prefix — CDN edge hit (~65% of traffic)
User types common prefix ('goo'). CDN edge cache hit. Sub-30ms end-to-end.
Client-side debounce. User types → wait 300ms after last keystroke → send query.
Optimize: Debounce is CRITICAL. Without it, user typing 'google' sends 6 queries. With debounce, 1 query for 'google'.
Small payload, HTTP/2 keep-alive.
Popular prefixes ('goo', 'ama') cached at edge with 30s TTL.
Small JSON response with top-10 suggestions.
Cache-hit path is **~30ms p99** (excluding client debounce). Sub-20ms server-side budget met with edge caching. **Debounce is the single most important optimization** — 6x fewer requests, better UX, less server load.
Bottleneck summary
Autocomplete latency is **DOMINATED BY CLIENT DEBOUNCE** (300ms — always). Server-side is 30-200ms depending on cache hit. **Amara paper 100ms = 1% revenue lost is the reference metric**. FST data structure (5-10x smaller than Trie) enables sub-10ms server-side lookups. Edge caching for popular prefixes + FST server for tail = balance between hit rate and freshness.
Optimization tips (this architecture)
- **Client-side debounce (300ms)**: CRITICAL. Without it, 6x more requests. With it, better UX + less load.
- **Edge cache popular prefixes (30s TTL)**: 65%+ hit rate. Sub-30ms end-to-end.
- **FST vs Trie**: 5-10x smaller in memory. Fits 1M+ suggestions in <1GB RAM per shard.
- **Flink 60s trending overlay**: Real-time trending queries blend with base suggestions. Refresh every 15s.
- **Prefix hash sharding**: Route by prefix hash to specific FST shard. Even distribution.
- **Personal L1 cache for premium**: Recent queries per user for personalization. Bypass edge cache.
- **HTTP/2 keep-alive**: Persistent connection eliminates handshake per keystroke.
- **Small payloads**: JSON responses <2KB. Compression optional (already small).
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.