Skip to main content
Back to Distributed Job Scheduler
MASTERCLASS
Gold-standard deep dive

Distributed Job Scheduler — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001
Architecture Decision Record

Temporal workflow orchestration + Kafka priority topics + Postgres durable state + leader-elected scheduler (etcd) + exactly-once activities via idempotency keys + dedicated observability (Airflow UI-equivalent) — over Linux cron, Kubernetes CronJobs, Airflow, or AWS EventBridge Scheduler + Step Functions.

Context
We are building the scale-out cron infrastructure for a company at 1M scheduled jobs/day. Job types: nightly billing runs (must NEVER double-charge), hourly analytics rollups, daily database backup, quarterly compliance reports, per-tenant scheduled webhooks. Critical constraint: EXACTLY-ONCE execution. Nightly billing must run — but must NOT run twice (double-charge = lawsuit). Reference deployments: Uber runs ~5M jobs/day on Cadence (now Temporal). Stripe uses Kafka + custom for billing runs. Airbnb uses Airflow but hit scaling ceiling at 100K jobs/day. The industry converged on Temporal for exactly-once workflow semantics.
Constraints
  • 1M scheduled jobs/day baseline, 10M/day peak (month-end + quarter-end billing runs)
  • Exactly-once semantics REQUIRED for financial jobs (billing, invoicing, payouts)
  • SLA: 95% of jobs run within 60 seconds of scheduled time; 99.9% within 5 minutes
  • Cross-team dependencies: ETL job A must complete before ML training job B (workflow)
  • Failure semantics: transient failures retry with exponential backoff (5 attempts max)
  • Auditability: every job execution logged with input + output + duration + success/failure
  • Team: 25 engineers total, 8 on Data Platform team
  • Existing infrastructure: 3-node Airflow (2 years old, at ceiling), Postgres, Kafka
Options considered

Temporal workflow orchestration + Kafka + Postgres + etcd leader election (chosen)

Pros
  • Exactly-once semantics via activity idempotency keys — billing-safe
  • Durable workflow state — worker crash mid-execution resumes from last checkpoint
  • Uber Cadence + Temporal battle-tested at 5M+ jobs/day
  • Kafka priority topics for job scheduling (urgent billing > nightly analytics)
  • etcd leader election prevents duplicate scheduler triggers
  • Cross-team workflow orchestration (ETL A → ML B → Deploy C) as first-class primitive
Cons
  • Temporal is new tech (~3 months team learning curve)
  • 3 stateful surfaces (Kafka + Temporal + Postgres) — dedicated Data Platform SRE required
  • Airflow migration 12-18 months (dual-run + gradual cutover)
  • Temporal SDK language-specific (Go, Java, Python, TypeScript)
Cost: $50K/mo infrastructure + $3M/yr Data Platform team

Continue Airflow (scale up)

Pros
  • Team knows Airflow
  • No migration risk
  • Web UI + DAG visualization built-in
Cons
  • Airflow scheduler is centralized — bottlenecks at 100K+ jobs/day
  • No exactly-once — Airflow uses at-most-once semantics (billing can be missed)
  • DAG-based orchestration is rigid — dynamic workflows hard
  • Airflow scheduler restart during nightly billing = missed run
Cost: $30K/mo but architecturally ceilinged

Kubernetes CronJobs

Pros
  • Native to K8s (we already run K8s)
  • Container-based (isolated per job)
  • Simple config (YAML cron expressions)
Cons
  • K8s CronJob doesn't guarantee exactly-once — can spawn duplicates on scheduler restart
  • No workflow orchestration (chained jobs impossible without external tool)
  • Monitoring is per-pod; no aggregate 'did nightly billing run?' dashboard
  • Concurrency policy (Allow/Forbid/Replace) is per-cronjob, not per-run
Cost: $5K/mo but unsafe for billing

AWS EventBridge Scheduler + Step Functions (managed)

Pros
  • Zero ops burden — AWS handles all scaling
  • EventBridge Scheduler: 1M+ scheduled events/day, per-event routing
  • Step Functions: workflow orchestration with retry, error handling
Cons
  • EventBridge Scheduler: $1 per 1M events × 1M/day × 30d = $30K/mo baseline
  • Step Functions: $0.025 per state transition × billions/mo = $50K+/mo
  • AWS lock-in on foundational scheduling infrastructure
  • Limited to AWS-supported activities (Lambda, SNS, ECS, etc.)
Cost: $100K+/mo + severe AWS lock-in

Custom scheduler on Postgres + advisory locks

Pros
  • Team already knows Postgres
  • Simple mental model (SQL + advisory locks for leader election)
Cons
  • Postgres advisory locks are per-database, not per-cluster — need pg_bouncer coordination
  • No workflow semantics — chained jobs require custom code
  • Re-inventing Temporal poorly
  • Ops burden owns everything (custom code = custom maintenance)
Cost: $10K/mo + 4 engineers dedicated to maintaining custom code
Chosen solution

Temporal workflow orchestration + Kafka + Postgres + etcd leader election + Airflow UI-equivalent observability

Why
The FIVE constraints that only Temporal satisfies: (1) exactly-once semantics for billing — Temporal's activity idempotency keys are designed for this, and it's the ONLY option that provides this without custom code. (2) Durable workflow state — nightly billing across 200 customer segments is a 4-hour workflow that MUST survive worker restarts. Temporal checkpoints activity results in Postgres. (3) Cross-team workflow orchestration — ETL A → ML B → Deploy C is a first-class primitive in Temporal, not a bolt-on. (4) Scale — Uber runs 5M+ jobs/day on Temporal's parent Cadence. We're at 1M/day; runway to 10x. (5) Auditability — Temporal's workflow history + activity replay lets us reconstruct 'what happened during nightly billing on Feb 15' at any point in time. Airflow ceilings at 100K jobs/day AND has no exactly-once (billing safety impossible). K8s CronJobs can duplicate on scheduler restart (billing safety impossible). AWS EventBridge is $100K+/mo AND severe AWS lock-in. Custom scheduler on Postgres is 'reinvent Temporal poorly'. Temporal is 'the answer' for billing-safe scheduled work — accept the 12-18 month Airflow migration.
Rejected alternatives (with reasons)
  • Airflow scaled up — centralized scheduler ceilings at 100K jobs/day + no exactly-once (billing safety broken)
  • Kubernetes CronJobs — duplicate spawns possible on scheduler restart = billing safety broken
  • AWS EventBridge + Step Functions — $100K+/mo + severe AWS lock-in on foundational infrastructure
  • Custom scheduler on Postgres — reinventing Temporal poorly + 4-engineer maintenance burden
Trade-offs accepted
  • Accept Temporal learning curve (~3 months for team to be productive with SDK)
  • Accept 3 stateful surfaces (Kafka + Temporal + Postgres) — dedicated Data Platform SRE required
  • Accept 12-18 month Airflow migration (dual-run + gradual DAG cutover)
  • Accept Temporal SDK language-specific lock (Go/Java/Python/TypeScript — no shell scripts)
  • Accept observability tooling investment (Airflow's UI is a strength; Temporal's is weaker, build custom dashboards)
  • Accept idempotency-key discipline — every activity must correctly handle 'was I already run?'
  • Accept Temporal ops complexity — dedicated 8-engineer Data Platform team owns billing-safety-critical infra
Consequences
  • Nightly billing becomes bulletproof — Temporal replay lets us reconstruct any billing run's exact execution
  • Cross-team workflows become simple to author (ETL A → ML B → Deploy C in 20 lines of Temporal SDK)
  • Long-running jobs (4-hour billing) survive infrastructure failures automatically
  • Exactly-once semantics propagate through downstream systems (payment processor, ledger, notifications)
  • Airflow expertise team becomes Temporal expertise team over 18 months — investment in future-facing skill
  • Data Platform team owns billing-critical infrastructure — heightened SLA + on-call standards
  • Compliance auditability improved 10x — 'did we run billing on this date?' answered from Temporal history
When would we reverse this decision?
  • If Temporal cluster ops cost exceeds 4 engineers dedicated → managed Temporal Cloud at 2x infrastructure cost
  • If job volume drops below 100K/day sustained → simplify back to Airflow (accept regression)
  • If AWS Step Functions matures to L6-scale exactly-once semantics → managed migration attractive
  • If competing exactly-once workflow engines (Restate, Dapr) emerge with better economics → evaluate at 3-year checkpoint

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are Data Platform lead at a $200M ARR SaaS company. CFO drops in Monday: 'Compliance auditor found we missed running our SOC 2 nightly backup twice in Q3 (Airflow scheduler restarted during the run). We have 6 months to fix this before next SOC 2 audit or we lose the certification and 3 largest customers ($40M ARR). Options: (1) migrate to Temporal for exactly-once, (2) add redundancy to Airflow (multi-scheduler), (3) buy AWS EventBridge Scheduler + Step Functions, or (4) hire a compliance auditor to argue the missed runs weren't material. Board meets Wednesday.'

Constraints
  • 1$200M ARR SaaS company, SOC 2 Type II certified (revenue-critical)
  • 23 largest customers ($40M ARR combined) require SOC 2 as contract requirement
  • 36-month deadline before next audit
  • 4Data Platform team: 8 engineers
  • 5Current infrastructure: Airflow 2.x (3 years old), Postgres, Kafka (used for other things)
  • 6Compliance-required jobs: 12 nightly (backup, security scan, audit log rotation), 30 hourly (metrics rollups)
  • 7Missing a compliance job = immediate SOC 2 finding at next audit
  • 8Board risk-tolerance: high for cost, LOW for compliance
Your question

Recommend to board Wednesday. Show cost, risk, and 6-month feasibility. Address CFO's implicit hope: 'can we just add redundancy to Airflow and skip the migration?'

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT
Nightly billing double-charge — Airflow scheduler failover triggered duplicate DAG run + exactly-once semantics failure

PagerDuty alert at 6:34am PT. Finance team reports 340 customers received DUPLICATE invoices for their monthly billing. Total impact: $8.2M in duplicate charges across the customer base. Root cause under investigation: Airflow scheduler failed over at 2:03am during the nightly billing DAG execution. On-call: 'billing dag ran twice, we need to refund immediately and understand why exactly-once semantics failed'.

Metrics
  • customers_affected_by_duplicate_charge: 340
  • duplicate_charge_total_dollars: 8,200,000
  • affected_customer_arr_at_risk_percent: 35 (of 340 customers, ~120 may churn from trust erosion)
  • airflow.scheduler.failover_events_last_24h: 1 (at 2:03am)
  • airflow.dag.billing_nightly.execution_count: 2 (expected: 1)
  • airflow.dag.billing_nightly.execution_duration_min: run1=4h, run2=4h
  • billing_service.transactions_created_last_24h: 660 (expected 340 = 2x)
  • payment_processor.charges_created_last_24h: 660 (all successful)
  • refund_processing_backlog: 340 refunds queued (~$8.2M)
  • customer_support_ticket_rate: 89/hour (baseline 4)
  • media.mentions.negative: 12 (Reddit, Twitter — 'they charged me twice!')
Logs
  • 02:00:00 airflow-scheduler-primary: 'starting billing_nightly DAG execution'
  • 02:03:14 airflow-scheduler-primary: 'SIGTERM received (kubernetes pod eviction — node maintenance)'
  • 02:03:15 airflow-scheduler-secondary: 'leader election triggered'
  • 02:03:47 airflow-scheduler-secondary: 'promoted to leader — scanning for missed DAG runs'
  • 02:03:52 airflow-scheduler-secondary: 'DAG billing_nightly.2024-02-15 has PENDING status — kicking off execution'
  • 02:03:53 airflow-scheduler-secondary: 'NOTE: DAG was already running under scheduler-primary before failover — this is a duplicate run'
  • 06:07:00 airflow-scheduler-secondary: 'billing_nightly.2024-02-15 completed (run 2)'
  • 06:07:14 finance-service: '340 duplicate invoices created for 2024-02-15'
  • 06:34:23 pagerduty: 'ALERT: customer support tickets rate spike + finance flags'
Dependency health
  • Airflow: DEGRADED — schedulers running but semantics broken
  • Billing service: HEALTHY (was doing what it was told — twice)
  • Payment processor (Stripe): HEALTHY — charged customers as instructed
  • Postgres: HEALTHY — durable state fine
  • Kafka: HEALTHY
  • Notification service: DEGRADED — sending 340 refund emails now
  • Customer support: OVERWHELMED — 89 tickets/hour
Your investigation
1

It's 6:47am. 340 duplicate charges, $8.2M impact. Customer support drowning. Finance panicking. What's your SINGLE highest-priority action in the next 15 minutes?

Hint: The refund process for 340 customers is going to take hours. What can you do RIGHT NOW to stop the customer trust bleeding?
2

Assume proactive email sent at 7:15am. Now: HOW did exactly-once semantics fail? Airflow's scheduler-primary failure during DAG execution triggered scheduler-secondary to re-run the DAG. Is this a bug in our Airflow config, a bug in Airflow itself, or a fundamental Airflow limitation?

Hint: Look at Airflow's execution model. When scheduler-primary is running a DAG and fails, what does Airflow's protocol say about the DAG's state?
3

It's 8:30am. Refunds are being processed (~$8.2M total, will take 3-5 business days per Stripe). Emails sent. Support ticket volume dropped. What's your action to prevent this specific class of incident from recurring in the NEXT 30 days (before Temporal migration completes)?

Hint: Temporal migration is 12-18 months out. What can we do in 30 days to make Airflow safer for exactly-once jobs?
4

Postmortem the next day. Name 3 action items ranked by impact reduction for future scheduler-failover events (before Temporal migration completes in 12 months).

Hint: Some are technical, some are process, some are architectural. Prioritize by risk-adjusted impact.
5

Draft the customer-facing status page update at T+3 hours (9:34am — after proactive emails sent + refunds queued but not yet processed). Constraint: honest about the technical cause, don't over-promise, address the trust erosion.

Hint: The customer email covered the individual impact. The status page is for the broader public — potential customers researching, media, existing customers looking for details.
Knowledge graph

Learn these first

  • Cron scheduling + leader election (etcd/ZooKeeper)
  • Exactly-once semantics via idempotency keys + Temporal workflow replay
  • Workflow orchestration for chained scheduled jobs
  • Airflow vs Temporal architectural comparison

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.