Task Scheduler: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Urgent task — payment webhook (~10% of traffic, sub-500ms budget)
Payment webhook received. Retry logic requires processing within 500ms or Stripe retries. High priority queue.
Kafka producer, acks=all for durability. Priority topic serves urgent workers.
Standard Kafka commit.
Dedicated worker pool for urgent queue. Poll interval very short (50ms).
Optimize: Urgent workers over-provisioned to keep queue empty. Cost of idle vs cost of late.
Actual task execution.
Persist completion for durability.
Urgent task enqueue-to-complete is **~100-500ms p99**. **Meets Stripe webhook 500ms budget**. Key: dedicated worker pool + short poll + over-provisioning. If urgent queue backs up = SLA breach.
Bottleneck summary
Task scheduler latency depends **entirely on queue priority + task type**. Urgent tasks: sub-500ms (dedicated over-provisioned pool). Bulk tasks: hours (throughput-optimized, cost-optimized). Workflows: minutes to hours (multi-step). **The critical architectural insight**: Temporal's durability + workflow semantics + Kafka's priority topics enable different SLAs on same infrastructure. This is why every mature company runs Temporal or similar.
Optimization tips (this architecture)
- **Dedicated worker pools per priority tier**: Prevents bulk-blocks-urgent.
- **Over-provision urgent workers**: Empty queue is good — cost of idle < cost of late.
- **Temporal for workflow semantics**: Durable state + resumption. Non-negotiable for multi-step tasks.
- **Kafka acks=all for critical tasks**: Data loss unacceptable.
- **Bulk tasks off-peak**: Reindex at 3am, not 3pm. Save infra cost.
- **Task token idempotency**: Same task ID = same result. Prevents duplicate processing.
- **Dead-letter queue**: Failed tasks go to DLQ. Manual review + replay.
- **Priority queue starvation monitoring**: If bulk queue never drains, investigate.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.