Kafka-like Streaming Platform: Junior → Architect Evolution
The mandated interactive flow. Step through each rung, ask what breaks FIRST, weigh the options, and defend the chosen architecture. This is how architectural thinking is learned — not by reading a fixed design, but by tracing how it evolves under growth pressure.
Scale Evolution Timeline
Step through 6 architectural rungs, from 10K RPS to 1B RPS. At each rung, ask: what will break FIRST? Why? What options exist? Which one do we pick — and what are we accepting?
This is the reasoning cycle that separates a Junior developer ("here's an architecture") from an Architect ("here's why this architecture, why now, and what breaks next").
10K RPS — in-process pub/sub or SQS
1. Current architecture
Where we are before growth pressure
3 × c6i.large app instances. Cross-service events flow through SQS queues (one per event type). Producer: `sqs.SendMessage`. Consumer: long-poll `sqs.ReceiveMessage`. Retention: 14 days (SQS max). Dead-letter queue for failed messages.
2. Growth trigger
What changed — the traffic/data force
Series A startup. 8 microservices, ~10K events/sec at peak. Event types: user-signed-up, order-placed, payment-succeeded, notification-queued. Producers publish → consumers process → next stage.
3. Bottleneck — what breaks FIRST?
The component that saturates as growth arrives
None yet — SQS is fine
SQS handles up to 3K TPS per standard queue. Multiple queues (one per event type) parallelize. Per-message cost is $0.40 per million — 10K events/sec × 86400s × 30d = 26B events/mo × $0.40/M = $10K/mo. High but not unreasonable for a Series A. p99 latency: 50-200ms (SQS long-poll + processing).
SQS TPS 8K sustained (well under 12K aggregate ceiling across queues), messages_in_flight <2K, DLQ rate <0.01%
4. Options — what could we do?
Alternatives an architect must consider before picking
- + Zero new infrastructure to operate
- + SQS visibility timeout + DLQ pattern is well-understood
- + Per-queue scaling is easy — add queues as event types grow
- + Team already has SQS ops experience
- − $10K/mo scales linearly with volume — will hurt at 10x
- − SQS has 14-day retention max — cannot serve replay-for-debugging use case
- − No exactly-once semantics (SQS FIFO has them but with 300 TPS limit)
- − No stream processing primitives (must build stateful workers from scratch)
5. Chosen
The specific decision we're making
Stay on SQS — Kafka would be premature at 10K events/sec
6. Trade-offs
What we're explicitly accepting to move forward
- Accept 14-day retention limit — no long-term event history
- Accept SQS cost trajectory — will migrate to Kafka around 100K events/sec
- Accept no stream processing — batch analytics runs nightly against Postgres or S3 exports
- Accept SQS 200ms tail — some events see 500ms in-flight time under load
7. New architecture
The system after this decision — headroom for the next 5-10x
8 microservices, ~30 SQS queues (one per event type), auto-scaled consumer workers on ECS. DLQ per queue. Total infra cost: $12K/mo (SQS + consumer workers).
8. Next bottleneck — what will break at the NEXT rung?
This is the seed of the next rung
At ~100K events/sec, SQS cost climbs past $100K/mo AND 14-day retention limit blocks the analytics team from replaying event history. Also stream processing (fraud detection, real-time aggregations) requires more than 'consume + write to DB'. Need Kafka — L5 shape.
What comes next in your Junior → Architect journey
You've traced 6 rungs of Kafka-like Streaming Platform evolution. Now try the same reasoning cycle on a system you don't know yet — pick from the Systems catalog and answer the same questions: current arch → growth trigger → bottleneck → options → chosen → trade-offs → new arch → next bottleneck. That is architecture.