Real-time Analytics: Junior → Architect Evolution
The mandated interactive flow. Step through each rung, ask what breaks FIRST, weigh the options, and defend the chosen architecture. This is how architectural thinking is learned — not by reading a fixed design, but by tracing how it evolves under growth pressure.
Scale Evolution Timeline
Step through 6 architectural rungs, from 10K RPS to 1B RPS. At each rung, ask: what will break FIRST? Why? What options exist? Which one do we pick — and what are we accepting?
This is the reasoning cycle that separates a Junior developer ("here's an architecture") from an Architect ("here's why this architecture, why now, and what breaks next").
10K events/sec — nightly batch (Spark)
1. Current architecture
Where we are before growth pressure
Nightly Spark job scans yesterday's data. Delivers dashboard next morning.
2. Growth trigger
What changed — the traffic/data force
Analytics MVP. Reports delivered 24h behind.
3. Bottleneck — what breaks FIRST?
The component that saturates as growth arrives
None yet — batch is fine at MVP
10K events/sec = ~1B/day. Nightly batch handles.
batch runtime 4h, data lag 24h
4. Options — what could we do?
Alternatives an architect must consider before picking
- + Simple
- + Cheap
- − 24h data lag
5. Chosen
The specific decision we're making
Do nothing — batch is right at MVP
6. Trade-offs
What we're explicitly accepting to move forward
- Accept 24h data lag
- Accept nightly batch model
7. New architecture
The system after this decision — headroom for the next 5-10x
Spark nightly batch. $500/mo.
8. Next bottleneck — what will break at the NEXT rung?
This is the seed of the next rung
At ~100K events/sec + product wants real-time dashboards (sub-1s queries), batch insufficient. Lambda architecture.
What comes next in your Junior → Architect journey
You've traced 6 rungs of Real-time Analytics evolution. Now try the same reasoning cycle on a system you don't know yet — pick from the Systems catalog and answer the same questions: current arch → growth trigger → bottleneck → options → chosen → trade-offs → new arch → next bottleneck. That is architecture.