Skip to main content
Back to Kafka-like Streaming Platform
MASTERCLASS
Gold-standard deep dive

Kafka-like Streaming Platform — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001
Architecture Decision Record

Immutable append-only log + partition-per-topic + configurable replication (ISR-based) + tiered storage (SSD hot + object-storage cold via KIP-405) — over pure DB-based event log, pure in-memory pubsub, or pure serverless dispatch

Context
We are building a durable event bus for a microservices architecture at scale (LinkedIn at 7T messages/day, Uber at 4T, Netflix Keystone at 15B events/day are the reference deployments). Consumers include real-time services (fraud detection, matching), batch analytics (Spark/Flink), ML training pipelines, and downstream microservices. Producer diversity: hundreds of microservices publish; hundreds of consumers subscribe. Retention requirements vary: 24-hour retention for hot events; 30-90 day retention for compliance-critical events; multi-year retention for some ML training data.
Constraints
  • 1M-7T messages/day at scale (LinkedIn's scale is the industry reference)
  • Throughput: 100K-1M messages/sec/topic at peak
  • Latency: producer-to-consumer p99 < 100 ms; typical 10-30 ms
  • Durability: acknowledged messages MUST NOT be lost (production of financial + compliance events)
  • Availability: 99.99% for producer-writes; 99.99% for consumer-reads
  • Retention: 24 hours to years, per-topic configurable
  • Consumer groups: 10s-100s per topic; each independently tracks offsets
  • Cross-region: some events replicate globally (analytics); some stay regional (compliance)
Options considered

Kafka-style immutable log + partition-per-topic + ISR replication + tiered storage (chosen)

Pros
  • Immutable log = highest write throughput possible on commodity hardware (append-only I/O is faster than random)
  • Partition-per-topic = horizontal scalability + per-partition ordering guarantee
  • ISR (in-sync replicas) replication + configurable acks (0/1/all) = trade durability vs. throughput per topic
  • Zero-copy reads via sendfile() syscall = broker CPU efficiency
  • Consumer groups + offset tracking = independent consumers without coordination overhead
  • KIP-405 tiered storage (SSD hot + S3 cold) = infinite retention at object-storage costs
  • Battle-tested at hyperscale (LinkedIn 7T/day, Uber 4T, Netflix Keystone 15B/day)
Cons
  • Broker operational burden — Kafka clusters have well-known operational quirks (leader election, ISR shrink, controller failover)
  • Partition-count decisions are quasi-permanent — hard to reshard without downtime
  • Consumer-lag monitoring is a first-class discipline; ignored lag = cascading backlog
  • Zookeeper dependency (though KRaft mode is replacing this in newer versions)
Cost: Kafka cluster + Kafka Connect + Kafka Streams: ~$200K/mo at 1T messages/day. Managed alternatives (Confluent Cloud, MSK) cost 2-4× that.

DB-based event log (Postgres, Cassandra) with pub/sub triggers

Pros
  • Familiar SQL + existing DB operations
  • Transactional integrity — event write is atomic with business logic
Cons
  • DB write throughput is 10-100× lower than Kafka append-log
  • Random-write patterns invalidate DB buffer pool at high volume
  • No native consumer-group semantics; must build them
  • Pub/sub triggers add DB CPU load; scaling issue at high throughput
Cost: Cheap upfront but catastrophic at scale.

AWS SQS + SNS (pure serverless)

Pros
  • Zero infrastructure to manage
  • Auto-scales with traffic
Cons
  • SQS: no ordering guarantees; visibility timeout instead of offset tracking; per-message billing at scale is expensive
  • SNS: fan-out but not durable long-term storage
  • Retention: SQS max 14 days; cannot serve batch-analytics + ML training use cases
  • No zero-copy reads; per-message CPU on the queue tier
Cost: $0.40 per million messages at 1T/day = $12K/day = $360K/mo. Expensive at scale.

Buy Confluent Cloud (managed Kafka)

Pros
  • Zero operational burden
  • Fully managed by Kafka's original creators
  • Managed tiered storage, geo-replication, security
Cons
  • Cost: 2-4× DIY Kafka at Netflix/Uber scale
  • Vendor lock-in on critical infrastructure
  • Some advanced features (custom KIPs, custom serializers) require self-hosted
Cost: $500K-1M/mo at 1T messages/day. Prohibitive at largest scales; sensible below ~1B events/day.
Chosen solution

DIY Kafka-style with tiered storage (option 1)

Why
At LinkedIn/Uber/Netflix scale, DIY Kafka is 2-4× cheaper than managed. The operational burden is real but well-understood — Kafka SRE is now a first-class discipline with established tooling (Cruise Control for balancing, Confluent Control Center for monitoring). DB-based logs don't scale to trillion-message workloads. Serverless is prohibitively expensive at scale. Buy-Confluent is right for smaller companies but crosses over into expensive at the hyperscale reference point. Every hyperscale reference (LinkedIn, Uber, Netflix, Twitter, Meta all publicly discussed) chose DIY Kafka. Recent additions like KIP-405 (tiered storage) made the economics even better for long-retention topics.
Rejected alternatives (with reasons)
  • DB-based event log — 10-100× lower throughput; catastrophic at scale
  • SQS + SNS — expensive per-message + no ordering + short retention
  • Confluent Cloud — right for < 1B events/day; too expensive at hyperscale
  • Redis Streams — good for small workloads but not durable at Kafka scale
Trade-offs accepted
  • Accept Kafka SRE burden — hire specialists; adopt Cruise Control + Kafka Manager + Kafka Streams
  • Accept partition-count decisions are quasi-permanent — over-provision partitions upfront
  • Accept ISR + zookeeper (or KRaft) dependency — the complexity is manageable
  • Accept tiered-storage complexity — S3 recovery is slower than SSD; plan capacity accordingly
Consequences
  • Kafka SRE team becomes a first-class discipline
  • Every microservice team must understand consumer-group + offset-tracking semantics
  • Producer libraries standardize retry + idempotence + acks configuration
  • Kafka Connect + Kafka Streams become internal patterns
  • Consumer-lag monitoring becomes an SLI for every consumer
When would we reverse this decision?
  • If our throughput drops sustainably below 1B events/day, buy Confluent Cloud
  • If new managed alternatives emerge with feature parity + lower cost (Redpanda, WarpStream), evaluate
  • If our workload shifts to real-time-only (no batch), pure SQS might suffice for the critical path
  • If regulatory landscape mandates specific cloud provider hosting, hosted options may become mandatory

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the tech lead at a Series B SaaS company. Your CTO reads a Netflix blog post about their Keystone data platform (15B events/day) + Kafka's importance in modern architecture, and says 'we should build an internal event bus based on Kafka for our microservices.' You have 3 engineers, 12 weeks, and currently ~10K events/day (mostly REST calls between your 8 microservices). The CTO wants 'Kafka-based event-driven architecture' shipped in the next quarter.

Constraints
  • 112-week deadline
  • 23 backend engineers; none has operated Kafka in production
  • 3Current traffic: 10K events/day between 8 microservices via REST
  • 4Existing stack: Rails + Postgres + Redis + Sidekiq on AWS
  • 5Budget: dedicated event-bus infra < $8K/mo
  • 6CTO's ask: 'Netflix Keystone-style event-driven architecture'
  • 7Team is comfortable with REST + Sidekiq; no async-event-driven experience
Your question

What do you propose, and how do you scope 'Kafka-based event-driven architecture' honestly? Be specific about buy vs. build, actual need, and the CTO conversation.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT
Broker failure + consumer-lag cascade — analytics pipeline falls 6 hours behind

PagerDuty alert at 14:22 UTC. Kafka broker-3 (of 6) has failed hard (kernel panic). ISR for topics that had broker-3 as leader is shrinking. Consumer lag for the 'user-events' topic (main analytics topic) has climbed from 200 ms to 6 hours in 30 minutes. Downstream Flink jobs are stalling because they can't get fresh data. Business dashboards show 'data delayed'. You are on-call.

Metrics
  • kafka.brokers.online: 5 of 6 (broker-3 down)
  • kafka.topic.user-events.isr_size: 2 of 3 replicas for partitions where broker-3 was leader
  • kafka.topic.user-events.consumer_lag: 6 hours 14 min (baseline 200 ms)
  • kafka.controller.election_count_last_hour: 47 (partition leader elections)
  • kafka.topic.user-events.messages_per_sec_produced: 2.3M/sec (baseline 500K/sec — 4.6× normal because it's Black Friday)
  • kafka.topic.user-events.messages_per_sec_consumed: 480K/sec (barely elevated)
  • downstream.flink_jobs.processing_delay: 6 hours (was <1 sec)
  • downstream.dashboards.affected: 12 dashboards showing 'delayed data' banner
  • traffic.pattern: Black Friday morning; producer rate 4-5× normal
Logs
  • 14:00 UTC — normal operations; broker-3 CPU at 65%
  • 14:07 UTC — broker-3 kernel panic; hard reboot begins
  • 14:07 UTC — Kafka controller detects broker-3 down; begins leader election for affected partitions
  • 14:08 UTC — new leaders elected; ISR shrinks from 3 → 2 for affected partitions
  • 14:10 UTC — producers begin retrying failed writes; producer buffer fills
  • 14:15 UTC — Black Friday sales spike arrives; producer rate 500K → 2.3M/sec
  • 14:18 UTC — remaining brokers absorb 4.6× write load + broker-3's share
  • 14:20 UTC — consumer lag begins climbing as producers write faster than consumers can catch up
  • 14:22 UTC — PagerDuty alert fires
Traces
  • The cascade sequence:
  • Broker-3 dies → partitions where broker-3 was leader elect new leaders from remaining ISR replicas
  • Producers retry writes with idempotence enabled — all writes eventually land
  • BUT: Black Friday producer rate is 4.6× baseline AND we're now down to 5 brokers instead of 6
  • Per-broker load: (2.3M/sec / 5 brokers) = 460K/sec per broker (baseline: 500K/sec / 6 = 83K/sec)
  • Brokers are at ~500K/sec capacity ceiling — they're pinned
  • Consumer read throughput is now competing with producer write throughput for broker I/O
  • Consumers fall behind faster than they can catch up
  • The dashboards falling behind:
  • Flink jobs consume from user-events, aggregate, write to analytics DB
  • Flink was tuned for 500K/sec input; now sees 2.3M/sec + 6-hour backlog
  • Flink cannot catch up until either producer rate drops OR consumer parallelism increases
Dependency health
  • Broker-3: DOWN (rebooting)
  • Brokers 1, 2, 4, 5, 6: healthy but CPU-saturated (~90%)
  • Zookeeper (or KRaft controller): healthy
  • Producers (upstream microservices): healthy but experiencing higher latency
  • Consumer group flink-analytics: DEGRADED (6-hour lag)
  • Consumer group ml-fraud-detection: DEGRADED (2-hour lag; smaller topic)
  • Downstream dashboards: DEGRADED (delayed data warning)
  • Business KPIs: DEGRADED (delayed analytics = delayed business decisions)
Your investigation
1

You look at the metrics. What's the ROOT cause of the consumer lag?

Hint: It's NOT just the broker failure. What's the compound problem?
2

60 seconds to decide a mitigation. What do you do?

Hint: You need to either (a) reduce producer rate or (b) increase consumer/broker capacity. Which is faster?
3

Cluster scale + rate-limit works — consumer lag drops to <5 min within 40 min. What's your postmortem + top action items?

Hint: The proximate cause was broker failure + traffic spike. What's the systemic gap?
4

In the retro, someone says 'we should switch to Confluent Cloud so we don't manage brokers.' Is that the right response?

Hint: Ops cost vs. cloud provider cost.
5

How would you have caught this in staging?

Hint: What test scenarios simulate 'broker down + peak traffic'?
Knowledge graph

Learn these first

  • Immutable append-only log + partition-per-topic semantics
  • ISR (in-sync replicas) + acks configuration tradeoffs
  • Consumer groups + offset tracking
  • Zero-copy reads (sendfile syscall) + tiered storage (KIP-405)

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.