Kafka-like Streaming Platform: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Producer → broker → acks=all (~30% of writes, critical topics)
Producer publishes to critical topic (payments, orders). Broker requires acks from ALL in-sync replicas before returning success. Highest durability, higher latency.
Encode event object to bytes using schema registry. Avro binary is compact + fast. Schema resolution via Schema Registry (cached).
hash(key) mod num_partitions. Determines which partition (and thus broker) receives the message.
Same-AZ TCP hop. Producer maintains persistent connections to all brokers it writes to.
Optimize: Enable batching (linger.ms=10-50ms). Amortizes network overhead across many events. Trade latency for throughput.
Append to log segment file. Page cache absorbs writes; fsync deferred. Kafka's append-only log is sequential — much faster than random writes.
Optimize: Configure log.flush.interval.ms for fsync policy. Default is 'never' (rely on OS fsync). Set to 1000ms for balance of durability + latency.
Replicas pull from leader via fetch protocol. Leader waits for ACKs from ALL in-sync replicas before ACK to producer. Cross-broker latency dominates.
Optimize: Place replicas in same rack for lower latency (but higher blast radius). Cross-rack for durability (higher latency). Trade-off.
Success returned to producer with offset assigned.
acks=all is ~10-30ms p99 vs acks=1 at ~5ms. **Cost of durability is ~5x latency.** Choose acks=all for payments/orders (data loss unacceptable) and acks=1 for logs/analytics (occasional loss OK). Kafka lets you tune this PER-TOPIC — same cluster serves both use cases.
Bottleneck summary
Kafka latency is **BIMODAL BY DURABILITY CHOICE**: acks=all is 10-30ms p99 (waits for replica ACKs), acks=1 is 5-10ms p99 (fire-and-forget replicas). **This is the correct architectural tradeoff** — Kafka lets you tune per-topic based on data value. Consumer latency is dominated by BATCH POLLING (500ms max wait or immediate on load). End-to-end producer→consumer is 10-500ms depending on consumer poll interval.
Optimization tips (this architecture)
- **Choose acks per topic**: acks=all for payments/orders, acks=1 for analytics/logs. Same cluster serves both.
- **Batch producers**: linger.ms=10-50ms + batch.size=32KB. Amortize network cost across many events.
- **Compression**: snappy/lz4 reduce payload 3-5x. Cost is CPU — worth it for network-bound workloads.
- **Zero-copy consumer**: sendfile syscall bypasses user-space copy. Enable when possible.
- **Batch consumers**: fetch.min.bytes=1MB for throughput; keep lower for latency. Tune based on workload.
- **Replica placement**: same rack = lower latency + higher blast radius. Different rack = higher latency + safer. Trade-off.
- **ISR monitoring**: alert on replica lag > 5s. Under-replication risk.
- **Partition count**: too few = throughput bottleneck; too many = coordination overhead. Rule of thumb: 3-6 partitions per broker.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.