MASTERCLASS

Gold-standard deep dive

Task Scheduler — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001

Architecture Decision Record

Kafka priority topics + Temporal workflow orchestration + Postgres state store + separate worker pools per priority tier + at-least-once semantics with idempotent tasks + DLQ with alerting — over Celery + Redis (current), pure Airflow, SQS + Lambda, or AWS Step Functions.

Context

We are building the async work platform for a 200-microservice architecture at 1M tasks/hour peak. Task types: 200+ (payment webhooks, email sends, PDF generation, video transcoding, ML inference, ETL jobs, notification fanouts, database cleanup). Priority tiers: urgent (payment webhooks, sub-500ms budget), normal (emails, 30s OK), bulk (nightly reindex, hours OK). Workflows: video-upload → transcode → moderate → publish is 5 chained tasks each of which can fail and needs resumption. Reference: Uber built Cadence (later Temporal) for exactly this problem. LinkedIn uses Kafka + custom orchestration. Stripe uses Kafka + Sidekiq + custom retry logic. The industry converged on Kafka + Temporal-style workflow orchestration for L6+ workloads.

Constraints

1M tasks/hour baseline, 10M/hour peak (monthly billing run + Black Friday spikes)
200+ task types, 500K unique task instances active at any time
3 priority tiers with distinct SLAs: urgent (500ms), normal (30s), bulk (hours)
Workflows require durable state + resumption on worker crash
At-least-once semantics with idempotent task design — no duplicates for critical tasks (payments)
12-18 month migration budget from Celery + Redis to Kafka + Temporal
Team: 40 engineers total, 15 on Task Platform team
Existing infrastructure: Celery + Redis (2 years old), Kafka (used for events)

Options considered

Kafka + Temporal workflow orchestration + Postgres state (chosen)

Pros

Temporal: durable workflow state, resumption on failure, exactly-once semantics via idempotency
Kafka: priority topics per tier, 1M+ events/sec ingest, replayable
Uber Cadence + Temporal (Uber spin-off) battle-tested at hyperscale
Workflows: multi-step tasks (video → transcode → moderate → publish) with checkpoints
Long-running tasks: workers can crash + resume from last state (no more 30-min transcoding lost)

Cons

Temporal is new tech (learning curve ~3 months for team)
3 stateful surfaces (Kafka + Temporal + Postgres) — dedicated Data Platform SRE required
Workflow semantics require rewriting existing tasks (12-18 month migration)
Temporal SDK ties us to specific language runtimes (Go, Java, Python, TypeScript)

Cost: $100K/mo (Kafka + Temporal cluster + storage) + $10M/yr Platform team

Continue Celery + Redis (scale up)

Pros

Team knows Celery
No migration risk
Zero new operational surfaces

Cons

Single Redis ceiling ~500K ops/sec — we hit this at 1M tasks/hour peak
No workflow semantics — chained tasks fail-and-restart from scratch
No exactly-once guarantee — duplicates possible on retry
Long-running tasks (30-min video transcoding) lose state on worker restart

Cost: $40K/mo but architecturally ceilinged

Airflow for workflows + Celery for one-off

Pros

Airflow handles DAGs + scheduling
Celery handles high-volume async

Cons

Airflow's centralized scheduler is bottleneck at 100K+ tasks/hour
Two systems to operate + coordinate
Airflow at 2024 vintage doesn't have exactly-once workflow guarantees

Cost: $60K/mo but incomplete

SQS + Lambda (managed)

Pros

Zero ops burden — AWS handles scaling
Auto-scale per queue

Cons

$0.40 per million SQS requests × 1M tasks/hour = $700/day = $250K/yr for messaging alone
Lambda 15-min max execution — video transcoding needs Fargate
Priority queues require multiple SQS queues + client-side routing (kludge)
AWS lock-in on foundational async infrastructure
No workflow orchestration (would need Step Functions on top = more cost)

Cost: $400K/mo at our scale + AWS lock-in

AWS Step Functions

Pros

Managed workflow orchestration
AWS-native integrations (Lambda, SQS, DynamoDB)

Cons

$0.025 per state transition × billions of transitions/mo = $50K+/mo
Vendor lock-in more severe than SQS+Lambda
Limited to AWS-supported activities

Cost: $500K/mo + AWS lock-in

Chosen solution

Kafka + Temporal workflow orchestration + Postgres state + separate worker pools per priority tier

Why

The 200-microservice architecture creates FIVE distinct requirements that only Kafka + Temporal satisfy: (1) 1M+ tasks/hour throughput (Kafka handles this trivially; Celery+Redis ceilings), (2) durable workflow state (video-upload → transcode → moderate → publish must survive worker crashes), (3) at-least-once semantics with idempotency (payments cannot double-charge — Temporal's activity idempotency is designed for this), (4) priority tiers with dedicated worker pools (urgent payment webhooks must not queue behind bulk reindexing), (5) long-running tasks with resumption (30-min video transcoding on worker restart). Temporal was purpose-built for this by Uber engineering team (originally Cadence, spun off as Temporal in 2019) — it's not just a workflow engine, it's the durable-state solution for async work. SQS + Lambda is $250K/yr just for messaging + AWS lock-in. Airflow's centralized scheduler bottlenecks at our scale. Continuing Celery is architecturally ceilinged. The Kafka + Temporal choice is 'the answer' at L6+ for async work — same shape as why every hyperscaler ended up with this pattern.

Rejected alternatives (with reasons)

Celery + Redis scaled up — architecturally ceilinged at 500K ops/sec + no workflow + no exactly-once
Airflow + Celery — centralized scheduler bottleneck at 100K+ tasks/hour, two systems to coordinate
SQS + Lambda — $250K/yr messaging alone + AWS lock-in + no workflows + 15-min Lambda limit
AWS Step Functions — $50K+/mo state transitions + severe AWS lock-in on foundational async infra

Trade-offs accepted

Accept 3 stateful surfaces (Kafka + Temporal + Postgres) — dedicated Data Platform SRE team
Accept 12-18 month migration from Celery + Redis (dual-run + gradual cutover)
Accept Temporal learning curve (~3 months for team to get productive)
Accept Temporal SDK language-specific lock (Go, Java, Python, TypeScript — not shell scripts)
Accept over-provisioned urgent worker pool (cost of idle < cost of late payment webhook)
Accept task idempotency discipline — every activity must handle 'was I already run?' correctly
Accept 15-20 engineer Task Platform team — foundational infrastructure warrants first-class ownership

Consequences

Task platform becomes horizontally-scalable — 1M → 10M tasks/hour requires more workers, not architecture change
Workflow patterns emerge as reusable primitive (payment saga, video upload, ML training pipeline)
Long-running tasks (30-min transcoding) survive worker crashes — massive reliability win
Priority tiers enforce SLAs — urgent payment webhooks NEVER wait behind nightly billing
Temporal + Kafka expertise becomes valuable — team can hire against this stack
DLQ + alerting discipline flows through all task types — no more silent failures
Migration from Celery generates 2 quarters of dual-run infrastructure cost — factor into budget

When would we reverse this decision?

If Temporal cluster ops cost exceeds 3 engineers dedicated → managed offering (Temporal Cloud) at 2x infrastructure cost
If task volume drops below 100K/hour sustained → simplify back to Celery + Redis (accept regression)
If AWS Step Functions matures to L6-scale (currently ceilinged) → managed migration becomes attractive
If competing workflow engines emerge (Restate, Dapr, etc.) with better economics → evaluate at 3-year checkpoint

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the Task Platform lead at a $100M ARR SaaS company. CTO says Monday: 'Temporal Cloud (Temporal's managed offering) costs $150K/yr. Self-hosted Temporal is 2 engineers-worth of ops overhead (~$400K/yr fully loaded). Celery + Redis works today but ceilings at 1M tasks/hour. We're at 400K tasks/hour now, projected 2M in 18 months. Which migration path: (1) migrate to Temporal Cloud (managed), (2) migrate to self-hosted Temporal, (3) stay on Celery + Redis until it breaks, or (4) hybrid — Celery for simple tasks + Temporal for workflows?'

Constraints

1$100M ARR SaaS company, 300 engineers, 15 on Task Platform
2Current: Celery + Redis, 400K tasks/hour, 30% workflow tasks (chained), 70% simple async
318-month growth projection: 2M tasks/hour (5x growth)
4Team can absorb 1 major migration or 2 minor migrations in next 18 months (not both)
5Existing pain: 5-10 workflow failures/week where video-transcoding jobs lose state on worker restart, 2 payment webhook duplicates/month, monthly billing runs taking 8 hours instead of 4
63-year budget: $2M available for platform investment
7Board wants 'foundational infrastructure' story for Series D pitch

Your question

Recommend the path Wednesday. Show 3-year TCO for each option + team capacity math + strategic risk. Address CTO's implicit question: 'is Temporal Cloud worth 2.7x self-hosted?'

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT

Bulk backlog cascade during monthly billing run — priority-tier starvation + Redis memory exhaustion + downstream service saturation

PagerDuty alert at 2:34am PT on the 1st of the month (monthly billing run). Bulk task queue depth 2.4M (baseline 5K). Urgent payment webhook latency spiked from 200ms to 45 seconds. Redis Cluster memory at 91% and rising. Downstream services (billing, notifications, ledger) reporting connection pool exhaustion. Marketing dashboard shows $840K/hour in delayed payment processing. On-call: 'billing run started 34 minutes ago, is 2 hours behind schedule, and taking down urgent tier'.

Metrics

task.bulk_queue_depth: 2,400,000 (baseline 5,000 = 480x)
task.urgent_queue_p99_latency_ms: 45,000 (baseline 200 = 225x)
task.urgent_success_rate: 68 pct (baseline 99.5 pct)
task.workers.urgent_pool_util: 12 pct (workers idle — but tasks stuck behind Redis backpressure)
redis.memory_utilization: 91 pct (rising ~1 pct/min)
redis.evictions_per_sec: 2400 (was 0 baseline)
task.bulk_success_rate: 45 pct (retry storm on downstream failure)
task.workers.bulk_pool_util: 100 pct (saturated)
downstream.billing_service.connection_pool_wait_p99: 8500ms (target <100ms)
downstream.billing_service.cpu_percent: 96
downstream.notification_service.connection_pool_wait_p99: 6200ms
delayed_payment_revenue_per_hour: $840,000

Logs

02:00:03 scheduler: 'monthly-billing-run kicked off (2M customer invoices)'
02:15:27 task-worker-bulk: 'bulk queue depth 500K, exceeds baseline threshold'
02:22:14 task-worker-bulk: 'billing service returning 5xx at 40 pct, entering exponential backoff'
02:28:00 task-worker-bulk: 'retry storm from backoff — bulk queue growing to 1.5M'
02:31:12 redis-cluster: 'memory 85 pct, LRU evictions engaged'
02:33:45 task-worker-urgent: 'urgent tasks queuing behind Redis backpressure — evictions affecting Redis latency'
02:34:03 payment-webhook-handler: 'PagerDuty alert — 6 payment webhooks timed out (Stripe will retry)'
02:35:18 downstream.billing-service: 'connection pool exhausted, rejecting new requests'
02:37:00 downstream.notification-service: 'connection pool exhausted, cascade beginning'
02:39:22 executive-slack: 'why is 50 pct of our platform down?'

Dependency health

Redis Cluster: DEGRADED — 91% memory, evictions engaged, latency affected
Task Workers (bulk pool): SATURATED — 100% utilization
Task Workers (urgent pool): IDLE-BUT-BLOCKED — 12% util, tasks stuck behind Redis backpressure
Billing Service (downstream): SATURATED — 96% CPU, connection pool exhausted
Notification Service (downstream): DEGRADED — connection pool wait 6+ seconds
Ledger Service (downstream): DEGRADED — cascade from billing
Payment Webhooks (Stripe): DEGRADED — 32% timing out, Stripe queuing retries
Postgres: HEALTHY at 55% CPU — not the bottleneck
Kafka: HEALTHY — event stream flowing

Your investigation

It's 2:39am. Bulk queue at 2.4M, urgent tier degraded, downstream services cascading. Payment webhooks timing out. What's your SINGLE highest-priority action in the next 5 minutes to stop the cascade?

Hint: The bulk retry storm is amplifying the problem. What can you change in the bulk consumer WITHOUT restarting anything?

Bulk queue stabilized at 2.4M (not growing anymore, thanks to disabled retry). But the queue is STILL there — 2.4M billing tasks. Urgent tier is degraded because Redis is at 91% memory. What's the highest-leverage action to drain Redis memory pressure?

Hint: Redis is memory-bound. The 2.4M bulk tasks each hold ~5KB of task state. That's 12GB of task state in Redis. What can you move OUT of Redis?

It's 3:15am — 41 minutes since alert. Redis stabilized, urgent tier recovered, bulk backlog draining from Postgres at 3K/sec. Billing service still degraded at 88% CPU. If we don't slow down bulk processing, billing service will re-cascade. What's your action?

Hint: You have manual control over bulk consumer rate. What's the right target rate?

Postmortem the next day. Name 3 action items ranked by impact reduction for future monthly billing runs.

Hint: Some are technical, some are process, some are architectural. Prioritize by risk-adjusted impact.

Draft the customer-facing status page update at T+90 minutes (4:04am — after urgent tier recovered but bulk backlog still draining). Customers are asleep for the most part, but B2B customers will see impact when they log in in the morning. Constraint: honest, no jargon, sets expectation for what they'll see when they wake up.

Hint: The visible customer impact is delayed monthly invoices + delayed notifications. Some payments will process 2-3 hours late. Set correct expectation.

Knowledge graph

Learn these first

Kafka priority topics + consumer groups
Temporal workflow orchestration + durable state + resumption
At-least-once semantics + idempotent task design
Priority tiers with dedicated worker pools (urgent vs bulk)

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.