Distributed Job Scheduler: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Recurring cron — critical job (~30% of jobs)
Nightly billing runs at 2am. Must execute exactly once. Temporal schedule + exactly-once activity.
Determine next execution time.
Confirm this scheduler node is leader (only leader triggers jobs).
Temporal receives workflow start request.
Persist workflow state for durability.
Worker picks up + executes.
Actual billing job — process 100K customers, generate invoices, submit charges.
Durable completion for exactly-once guarantee.
Schedule-to-execute is **~100ms + execution time**. **Exactly-once semantics come from idempotent activities + durable Temporal state**. If worker crashes mid-execution, Temporal resumes from checkpoint — no duplicate charges.
Bottleneck summary
Distributed job scheduler latency has TWO parts: **schedule overhead (~100ms — negligible)** + **execution time (seconds to hours)**. The execution time dominates. **The critical architectural insight**: Temporal + Kafka + exactly-once semantics enable scale-out cron with billing-safe guarantees. Without this, the pattern is 'fingers crossed cron'.
Optimization tips (this architecture)
- **Temporal for exactly-once**: Idempotent activities + durable state = billing-safe.
- **Leader election for scheduler**: Only leader triggers jobs. Prevents duplicates.
- **Exponential backoff on retry**: 1s, 5s, 25s, 2min. Cap retries. DLQ on max.
- **Job priority + separate worker pools**: Prevent bulk from blocking urgent.
- **Monitoring + alerting**: 'Did nightly billing run?' should be a dashboard, not SSH.
- **Idempotency keys per job type**: Same run = same result. No duplicate charges.
- **Duration limits per job**: Kill runaway jobs to prevent resource starvation.
- **Explicit retry policy per job type**: Financial jobs = no retry. Idempotent jobs = 3 retries.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.