Skip to main content
Back to Real-time Analytics
MASTERCLASS
Gold-standard deep dive

Real-time Analytics — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001
Architecture Decision Record

Kappa architecture (stream-only, single codebase) + Kafka event streams + Flink windowed processing + Druid (or Apache Pinot) OLAP database + pre-aggregated segments + columnar storage + real-time + historical segment seamless boundary — over Lambda architecture (batch + stream), pure Spark batch, Snowflake real-time (Snowpipe Streaming), or ClickHouse.

Context
We are building the real-time analytics platform at 1M events/sec sustained ingest, 10M/sec peak (Black Friday, product launches). Reference deployments: Uber's dispatching analytics runs on Flink + Pinot. Netflix Keystone processes 4M events/sec through Flink + Druid. Airbnb's Superset dashboards query Druid backend. LinkedIn Pinot serves 100K+ QPS on billions of rows. Query patterns: 90% interactive dashboards (sub-second p99 required), 5% ad-hoc exploration (2-10s acceptable), 5% real-time alerts on aggregate metrics (500ms budget). The Kappa architecture pattern (stream-only, single codebase) has won over Lambda (batch + stream, two codebases) in industry consensus since 2020.
Constraints
  • 1M events/sec baseline, 10M/sec peak
  • 500 TB of event data, 3-year retention
  • Query SLA: dashboard queries sub-second p99, ad-hoc 2-10s, alert eval 500ms
  • Ingest-to-queryable lag: <60 seconds p99 for real-time queries
  • OLAP queries: SUM/COUNT/AVG/percentile aggregations over 100M+ rows
  • Team: 20 engineers, 8 on Data Platform (currently split batch team + streaming team)
  • Cost bound: $60K/mo infrastructure budget for real-time analytics
  • Reprocessing: must be able to replay historical events for model retraining or backfill
Options considered

Kappa: Kafka + Flink + Druid (or Pinot) + S3 (chosen)

Pros
  • Single codebase (Flink) — no batch/stream drift
  • Druid/Pinot pre-aggregated segments = sub-second OLAP on billions of rows
  • Real-time queries hit Kafka indexer (last 5 min); historical queries hit Druid segments — seamless
  • Kafka as source of truth enables replayable reprocessing
  • Flink exactly-once semantics for critical aggregations
  • Uber, Netflix, Airbnb, LinkedIn all use Druid/Pinot for this pattern
Cons
  • Kappa reprocessing requires replaying from Kafka (slow for 3-year retention)
  • Druid ops complexity — coordinator, broker, historical, real-time nodes
  • 12-18 month migration from existing Lambda (batch + stream) architecture
  • Team must unlearn 'batch + stream' mental model
Cost: $40K/mo (Kafka + Flink + Druid + S3) + $8M/yr Data Platform team

Lambda architecture (batch Spark + streaming Storm/Flink)

Pros
  • Existing team knowledge
  • Batch layer provides accuracy backstop for stream errors
  • Standard industry pattern circa 2015
Cons
  • TWO codebases — batch + stream drift causes bugs (industry pain point)
  • Team split: batch team + streaming team = coordination overhead
  • Consistency issues: 'batch says X, stream says Y, which is right?'
  • Modern consensus (post-2020): Kappa wins on maintainability
Cost: $60K/mo (two systems to operate) but strategic debt

Pure Spark batch (no streaming)

Pros
  • Simplest architecture
  • Team knows Spark
  • Accuracy over speed
Cons
  • 24-hour data lag unacceptable for real-time dashboards
  • Cannot serve interactive queries — batch runs are minutes to hours
  • Product team wants real-time metrics; batch fundamentally can't provide
Cost: $10K/mo but product-blocking (24h data lag)

Snowflake real-time (Snowpipe Streaming)

Pros
  • Managed service — zero ops
  • SQL familiar interface
  • Auto-scaling storage + compute
Cons
  • Snowpipe Streaming latency 10-30 seconds (vs sub-5s for Druid)
  • Cost: $2 per credit × 100 credits/hour = $150K/mo at our scale
  • Snowflake vendor lock-in on foundational infrastructure
  • Interactive dashboard queries (sub-second) not achievable — Snowflake is 2-10s p99
Cost: $150K/mo but SLA mismatch

ClickHouse

Pros
  • Purpose-built for OLAP — sub-second aggregations
  • Simpler ops than Druid
  • SQL interface
  • Open source (Yandex)
Cons
  • Real-time ingest less mature than Druid (10-30s lag vs 1-5s)
  • Reprocessing story weaker (must dedupe manually)
  • Cluster ops still requires expertise (ZooKeeper coordination)
  • Less battle-tested at 1M events/sec than Druid/Pinot
Cost: $30K/mo but real-time ingest weaker

AWS Kinesis + Redshift Streaming

Pros
  • Managed AWS services
  • Kinesis Data Firehose + Kinesis Analytics
  • Redshift Streaming Ingestion
Cons
  • AWS lock-in on foundational analytics
  • Redshift Streaming latency 10-30 seconds
  • Kinesis: $0.015 per shard hour × 200 shards = $2K/mo Kinesis + expensive Redshift concurrency
  • Total cost $80-120K/mo
Cost: $80-120K/mo + severe AWS lock-in
Chosen solution

Kappa: Kafka + Flink + Druid (or Apache Pinot) + S3 for long-term storage

Why
The FIVE constraints that drive Kappa + Druid choice: (1) Sub-second dashboard queries on billions of rows — only pre-aggregated columnar OLAP databases (Druid, Pinot) can do this. Snowflake is 2-10s (too slow). ClickHouse is close but weaker real-time ingest. (2) Single codebase — Kappa architecture eliminates the batch/stream drift bug class that plagues Lambda. Netflix Keystone team wrote extensively about this pain (2020). (3) Real-time + historical seamless — Druid's real-time task (Kafka indexer for last 5 min) + historical segments create unified query interface. Users don't know which nodes serve their query. This is Druid's key architectural feature. (4) Exactly-once semantics via Flink — critical for financial aggregations (Sum of transactions today MUST be right). (5) Cost — $40K/mo self-hosted vs $150K Snowflake vs $80-120K AWS Kinesis+Redshift. At our scale, self-hosted wins. Industry consensus (Uber, Netflix, Airbnb, LinkedIn) has converged on Kafka + Flink + Druid/Pinot for real-time analytics at L6+. Choosing this architecture accepts 12-18 month migration cost + Druid ops complexity in exchange for future-proof foundation.
Rejected alternatives (with reasons)
  • Lambda architecture — two codebases (batch + stream) causes drift bugs, coordination overhead
  • Pure Spark batch — 24h data lag makes real-time dashboards impossible
  • Snowflake real-time — 10-30s latency + $150K/mo + vendor lock-in
  • ClickHouse — weaker real-time ingest maturity + reprocessing story
  • AWS Kinesis + Redshift Streaming — severe AWS lock-in + $80-120K/mo + latency mismatch
Trade-offs accepted
  • Accept Druid ops complexity — coordinator, broker, historical, real-time nodes = 4 role types
  • Accept 12-18 month migration from Lambda (dual-run + gradual DAG cutover)
  • Accept team learning curve on Flink + Druid mental models
  • Accept Kafka as source-of-truth discipline — every event MUST land in Kafka
  • Accept 3-year Kafka retention cost ($5K/mo) — enables reprocessing
  • Accept reprocessing time cost — replaying 3 years takes 6-12 hours (bounded by Kafka read throughput)
  • Accept 15-20 engineer Data Platform team investment — combined batch + streaming teams
Consequences
  • Single-codebase Flink jobs eliminate batch/stream drift class of bugs
  • Sub-second dashboards enable real-time product decisions
  • Real-time alerts on aggregate metrics catch anomalies within seconds
  • Kafka source-of-truth enables reprocessing for model retraining, backfills, migrations
  • Team merges (batch + streaming) into unified Data Platform team
  • Druid + Pinot expertise becomes competitive engineering skill
  • Cost trajectory stays flat at $40K/mo through 3x growth (vs vendor pricing that scales linearly)
When would we reverse this decision?
  • If sub-second dashboard SLA becomes less critical (batch-tolerant business shift) → Snowflake managed at 2-10s
  • If team drops below 4 engineers on Data Platform → managed Confluent Cloud + Snowflake worth 3x cost
  • If Druid maturity concerns emerge (community issues) → migrate to Apache Pinot (similar architecture)
  • If Snowpipe Streaming latency improves to <2s → managed migration attractive

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are Data Platform lead at a $150M ARR e-commerce company. Chief Data Officer says Monday: 'Product team wants real-time inventory dashboards (currently batch, 4-hour lag). Options: (1) migrate from Lambda to Kappa + Druid ($40K/mo + 12-month migration), (2) buy Snowflake real-time ($150K/mo + 3-month setup), (3) build custom on ClickHouse ($30K/mo + 9-month migration), or (4) upgrade our existing Spark batch to run every 15 minutes (mini-batch — quick fix, $15K/mo). CFO wants budget clarity by Friday.'

Constraints
  • 1$150M ARR e-commerce, growing 40%/yr
  • 2Current: Lambda architecture (Spark batch + Storm streaming, both 4-year-old codebases)
  • 3Data volume: 500K events/sec baseline, 5M/sec Black Friday peak
  • 48 engineers on Data Platform: 5 batch (Spark experts), 3 streaming (Storm experts)
  • 5Query SLA business demands: sub-second dashboards, real-time alerts
  • 6Existing pain: batch vs stream data drift bugs — 15 per quarter, 4 hours to root-cause each
  • 7Product team blocked: cannot build real-time features (inventory, personalization) with 4-hour data lag
  • 8Board risk-tolerance: high for cost savings AND product velocity, moderate for migration risk
Your question

Recommend to CFO Friday. Show 3-year TCO for each option + engineering effort + product velocity impact + risk. Address CDO's implicit question: 'is 12-month migration risk worth it vs 3-month Snowflake purchase?'

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT
Druid broker cascade during Black Friday flash-sale — real-time dashboards go dark while executives watch

PagerDuty alert at 6:00pm PT on Black Friday. Druid brokers (query routing layer) at 98% CPU, query timeouts propagating. Real-time revenue dashboards (executives watching live) showing 'no data' for 2 minutes. Downstream: alert rules failing, product team cannot see conversion metrics, marketing team cannot see attribution. Root cause investigation: Kafka consumer lag growing (0.5s → 45s) causing real-time task index build-up. This is the biggest sales day of the year with the biggest audience of executives.

Metrics
  • druid.brokers.cpu_percent: 98 (baseline 60)
  • druid.query.timeout_rate_percent: 40 (baseline 0.1)
  • druid.query.p99_latency_seconds: 15 (baseline 0.8)
  • druid.real_time_tasks.lag_seconds: 45 (baseline 0.5)
  • kafka.consumer.lag_seconds: 45
  • kafka.events_per_sec: 8,500,000 (baseline 500,000 = 17x — Black Friday spike)
  • dashboard.revenue_realtime.request_error_rate_percent: 78
  • dashboard.executive_watching_count: 47 (people on the real-time revenue dashboard right now)
  • black_friday_current_hour_revenue: $28M (context: this hour matters)
  • on_call.time_to_first_response_seconds: 47 (fast)
Logs
  • 17:55:00 kafka: 'events_per_sec crossing 5M — 10x baseline'
  • 17:56:14 druid-real-time-task: 'ingest lag 15s and growing'
  • 17:58:22 druid-broker: 'query CPU 85 pct'
  • 17:59:44 druid-broker: 'query CPU 95 pct — queries queueing'
  • 18:00:03 druid-broker: 'query CPU 98 pct — timeouts propagating'
  • 18:00:15 dashboard.executive: 'chart shows -- (no data for 30 seconds)'
  • 18:00:47 pagerduty: 'ALERT: Druid broker CPU + query timeout spike'
  • 18:00:52 executive-slack: 'why is revenue dashboard dark?'
  • 18:02:11 chief-revenue-officer: 'CEO is watching, we need answers'
Dependency health
  • Druid brokers: SATURATED — 98% CPU, timing out queries
  • Druid real-time tasks: DEGRADED — 45s ingest lag
  • Druid historicals: HEALTHY (historical segments unaffected)
  • Kafka: HEALTHY — flowing events, but at 17x baseline
  • Flink jobs: HEALTHY at 78% CPU (headroom)
  • Dashboard consumers (executives): BLOCKED — dashboards showing no-data
  • Alert rules: FAILING — cannot evaluate against Druid queries
Your investigation
1

It's 6:03pm. Executives (including CEO) watching a dark revenue dashboard on Black Friday. What's your SINGLE highest-priority action in the next 3 minutes?

Hint: Real-time dashboards are broken, but historical data (from 1 minute ago) is fine. What can you switch dashboards to?
2

Executives unblocked via historical-only fallback. Now: how do you recover Druid broker CPU (still 98%)?

Hint: Druid brokers are the query routing layer. What's making them work harder — is it query volume or per-query cost?
3

It's 6:35pm. Brokers scaled, rate limits in place, historical fallback working, executives seeing dashboards. But Kafka event rate still 8.5M/sec (17x baseline) and Druid real-time tasks are still lagging 45s. Do we intervene, or wait for the traffic surge to pass?

Hint: Kafka can buffer indefinitely. Real-time tasks can be scaled up. What's the trade-off between: (a) scale up to catch real-time, or (b) accept lag until traffic normalizes?
4

Postmortem the next day. Name 3 action items ranked by impact reduction for future high-traffic events (Cyber Monday next week, Christmas, next Black Friday).

Hint: Some are technical, some are capacity planning, some are process. Prioritize by risk-adjusted impact.
5

Draft the executive communication at T+90 minutes (7:30pm — after dashboards recovered but incident is documented). Constraint: honest about what happened, precise about impact, no jargon, respects the CEO's attention.

Hint: The CEO watched real-time dashboards go dark on Black Friday. Executive comms need to be brief, credible, and reassure that we understand what happened.
Knowledge graph

Learn these first

  • Kappa vs Lambda architecture trade-offs
  • Druid / Pinot OLAP columnar databases
  • Flink stream processing + exactly-once semantics
  • Pre-aggregated segments for sub-second interactive queries

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.