Task Scheduler: Junior → Architect Evolution
The mandated interactive flow. Step through each rung, ask what breaks FIRST, weigh the options, and defend the chosen architecture. This is how architectural thinking is learned — not by reading a fixed design, but by tracing how it evolves under growth pressure.
Scale Evolution Timeline
Step through 6 architectural rungs, from 10K RPS to 1B RPS. At each rung, ask: what will break FIRST? Why? What options exist? Which one do we pick — and what are we accepting?
This is the reasoning cycle that separates a Junior developer ("here's an architecture") from an Architect ("here's why this architecture, why now, and what breaks next").
10K RPS — Sidekiq + Redis (in-process worker)
1. Current architecture
Where we are before growth pressure
3 × c6i.large Rails apps + Postgres + Redis + Sidekiq workers (bundled in same processes as web tier). Cron via whenever gem or Rails scheduler.
2. Growth trigger
What changed — the traffic/data force
Early product. 10K tasks/hour: welcome emails, PDF generation, webhook retries. Cron: nightly database cleanup, hourly analytics rollups.
3. Bottleneck — what breaks FIRST?
The component that saturates as growth arrives
None yet — Sidekiq handles this
Sidekiq processes ~1000 tasks/second per worker. 3 workers × 1000 = 3000/sec, we need 3/sec sustained. Redis handles queue state trivially. Cron via whenever generates system crontab entries.
Sidekiq queue depth <100, task latency p99 300ms, Redis CPU 15%
4. Options — what could we do?
Alternatives an architect must consider before picking
- + Zero infrastructure to add
- + Sidekiq is battle-tested + free (Pro version optional)
- + Rails ecosystem — team knows it
- − Redis SPOF for queue state
- − No cross-region distribution
- − Cron on single server = if server dies, jobs miss
5. Chosen
The specific decision we're making
Do nothing — Sidekiq is genuinely right at MVP
6. Trade-offs
What we're explicitly accepting to move forward
- Accept Redis SPOF for queue
- Accept cron single-server risk (missed daily jobs on restart)
- Accept ceiling around 100K RPS on single Sidekiq cluster
7. New architecture
The system after this decision — headroom for the next 5-10x
3 Rails + Sidekiq + Redis + Postgres. Total cost: $400/mo.
8. Next bottleneck — what will break at the NEXT rung?
This is the seed of the next rung
At ~100K tasks/hour with priority differentiation (urgent vs bulk) + retry policies + delayed jobs, Sidekiq's single-queue model becomes limiting. Need Celery-style priority queues + dead-letter queue + separate cron infrastructure. L5 shape.
What comes next in your Junior → Architect journey
You've traced 6 rungs of Task Scheduler evolution. Now try the same reasoning cycle on a system you don't know yet — pick from the Systems catalog and answer the same questions: current arch → growth trigger → bottleneck → options → chosen → trade-offs → new arch → next bottleneck. That is architecture.