Distributed Logging System: Junior → Architect Evolution
The mandated interactive flow. Step through each rung, ask what breaks FIRST, weigh the options, and defend the chosen architecture. This is how architectural thinking is learned — not by reading a fixed design, but by tracing how it evolves under growth pressure.
Scale Evolution Timeline
Step through 6 architectural rungs, from 10K RPS to 1B RPS. At each rung, ask: what will break FIRST? Why? What options exist? Which one do we pick — and what are we accepting?
This is the reasoning cycle that separates a Junior developer ("here's an architecture") from an Architect ("here's why this architecture, why now, and what breaks next").
10K logs/sec — syslog + rsync + grep
1. Current architecture
Where we are before growth pressure
10 servers running syslog to local /var/log. Nightly rsync consolidates to central log server. Search: SSH + grep. Retention 30 days.
2. Growth trigger
What changed — the traffic/data force
Early product. 100K logs/day total. Debugging via SSH + grep.
3. Bottleneck — what breaks FIRST?
The component that saturates as growth arrives
None — syslog + grep is fine
10K logs/sec is nothing. Local disk stores easily. grep on 30 days of logs = 15 seconds.
log volume 10 GB/day, grep time 5-15s, storage cost $50/mo
4. Options — what could we do?
Alternatives an architect must consider before picking
- + Zero infrastructure
- + Team knows Linux tooling
- − Doesn't scale past 10 servers
- − No search across servers
5. Chosen
The specific decision we're making
Do nothing — syslog + grep is right at MVP
6. Trade-offs
What we're explicitly accepting to move forward
- Accept SSH-only debugging
- Accept ceiling at ~100 servers
7. New architecture
The system after this decision — headroom for the next 5-10x
10 servers × syslog. Total cost: $50/mo.
8. Next bottleneck — what will break at the NEXT rung?
This is the seed of the next rung
At ~100K logs/sec with 100+ services + microservices architecture, need centralized log aggregation with search. Elasticsearch + Kibana. L5.
What comes next in your Junior → Architect journey
You've traced 6 rungs of Distributed Logging System evolution. Now try the same reasoning cycle on a system you don't know yet — pick from the Systems catalog and answer the same questions: current arch → growth trigger → bottleneck → options → chosen → trade-offs → new arch → next bottleneck. That is architecture.