Real-time Analytics: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Event ingest — Kafka → Flink → Druid (~100% of throughput)
Event flows from app → Kafka → Flink transforms → Druid stores.
Kafka producer.
Stream processing.
Optimize: Windowing strategy affects latency. Tumbling windows for accuracy, sliding for smoothness.
Segment commit every ~5 min. Real-time only for last 5 min via Kafka indexer.
Event-to-queryable is **~5-60s in Druid** (segment commit). **Real-time queries** hit Kafka indexer for last 5 min + Druid segments for older data. This is the Kappa architecture pattern.
Bottleneck summary
Real-time analytics latency: **ingest ~5-60s to queryable**, **OLAP query sub-second on billions of rows**, **real-time query ~200ms-1s for last 5 min**. The **Kappa architecture** + Druid's real-time + historical segment boundary + columnar storage + pre-aggregation = sub-second interactive queries on massive data. This is why Uber, Netflix, Airbnb all use Druid/Pinot for real-time analytics.
Optimization tips (this architecture)
- **Pre-aggregation at index time**: Materialized rollups for common queries. 100x faster than raw scan.
- **Columnar storage (Druid/Pinot)**: Analytical queries scan columns, not rows.
- **Real-time + historical seamless**: Kafka indexer + Druid segments = unified query interface.
- **Kappa architecture**: Single codebase (Flink), reprocess from Kafka for changes.
- **Windowing strategy**: Tumbling for accuracy, sliding for smoothness.
- **Segment size tuning**: 500 MB typical. Smaller = faster query, more metadata overhead.
- **Retention tiers**: Hot 7d + warm 90d + cold S3 for older.
- **Materialized views**: Pre-compute for known dashboards.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.