Distributed Logging System: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Hot query — recent logs (last 24h) (~85% of queries)
Engineer searches logs from last 24h in Kibana. Data in hot SSD tier.
User query submitted.
Query planning.
Full-text search across shards.
Optimize: Time-based indices (daily) reduce shards to search.
Merge sorted results.
Rendered.
Hot query is **~500ms-1.5s p99**. Full-text search dominant. Time-based indices help — only search relevant days.
Bottleneck summary
Logging latency has two profiles: **hot query (~500ms-1.5s)** and **cold query (seconds to minutes)**. Ingest is ~100-500ms end-to-end. **The critical trade**: storage tiers optimize cost 10x but slow cold queries. Time-based indices + label-based routing (Loki) reduce query cost.
Optimization tips (this architecture)
- **Time-based indices**: Daily indices. Query only relevant days.
- **Tiered storage**: 7d hot + 90d warm + Glacier cold. 60 pct cost savings.
- **Kafka buffer**: Absorbs ingest spikes.
- **Logstash pipeline optimization**: Reduce enrichment overhead.
- **ES refresh interval**: 30s for high-volume (not default 1s).
- **Loki for cost-sensitive workloads**: Label-based indexing = 10x cheaper.
- **Warm-cache before compliance**: Pre-hydrate S3 tier.
- **Query result caching**: Repeated dashboard queries served from cache.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.