Skip to main content
Back to Food Delivery
MASTERCLASS
Gold-standard deep dive

Food Delivery — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001
Architecture Decision Record

Redis Cluster (32 shards) for driver location + Kafka event stream + ML dispatch service with prep-time prediction (DeepETA-style) + real-time surge pricing per hex + hybrid driver marketplace (exclusive scheduling + multi-app-aware scoring) + WebSocket for live tracking + async fulfillment via Sidekiq — over naive-nearest dispatch, PostGIS single-region, third-party Uber Direct integration, or Cloudflare Workers for edge dispatch.

Context
We are building the DoorDash/Uber Eats/Deliveroo pattern — 3-sided marketplace with 50 cities, 5000+ restaurants, 2000+ drivers, 1M order-related RPS peak (mostly driver location updates + browse traffic). Dinner rush is 30x baseline. Reference: DoorDash IPO'd at $58B with this architecture (2020); Meituan operates at 100M+ orders/day globally. Compound ETA: browse (fast) + order commit (Stripe-dominated) + restaurant acceptance (5s-2min unbounded) + food prep (10-30min) + driver dispatch (5-30s ML-scored) + delivery (10-30min) = 30-60 min end-to-end. The critical UX is HIDING the compound tail via progress-state UI — 'sending order' → 'accepted' → 'preparing' → 'driver assigned' → 'delivering'.
Constraints
  • 50 cities × 500K orders/city/month = 25M orders/mo; dinner rush 30x baseline
  • 5000 restaurants + 2000 drivers active per city
  • Match latency budget: <30s from order-accepted to driver-dispatched
  • Dispatch cancellation rate: <5% (currently 15% with naive-nearest)
  • 3-sided marketplace: customer + restaurant + driver — each has different SLAs
  • Real-time surge: driver-supply/customer-demand ratio per H3 hex, updated every 60s
  • Regulatory: California AB5 (gig worker classification), EU Platform Work Directive (2024), state-by-state marketplace tax
  • Multi-app driver reality: 40% of drivers work DoorDash + Uber Eats + Grubhub simultaneously — dispatch scoring must account for this
  • Team: 60 engineers total, 20 dedicated to dispatch platform
Options considered

Redis Cluster + Kafka + ML dispatch + WebSocket tracking (chosen)

Pros
  • Redis GEOADD/GEORADIUS for driver location — 100K+ writes/sec across 32 shards
  • ML dispatch predicts prep-time + delivery ETA + driver acceptance rate — reduces cancellation 15%→5%
  • WebSocket for live tracking = sub-second position updates to customer's phone
  • Kafka event stream decouples order lifecycle: OrderPlaced → RestaurantAccepted → DriverDispatched → PickedUp → Delivered
  • Async fulfillment via Sidekiq for email/warehouse notifications keeps request path fast
  • Multi-app-aware scoring: penalize driver acceptance score if they've been reject-happy on our app recently
  • DoorDash + Uber Eats + Deliveroo actual production pattern
Cons
  • Redis SPOF per shard — plan for shard failover during peak
  • ML dispatch requires GPU inference tier ($50K+/mo)
  • Real-time surge PR risk (customers complain about dynamic pricing) — be transparent
  • Kafka + Redis + Postgres + Sidekiq = 4 stateful surfaces to operate
  • WebSocket infrastructure = new discipline (connection management, reconnect storms)
Cost: $100K/mo infrastructure + $10M/yr Platform + ML + Dispatch team

Naive-nearest dispatch on PostGIS

Pros
  • Simplest architecture — one Postgres does everything
  • Team already knows PostGIS + Rails
  • Zero ML infrastructure
Cons
  • Ignores prep time — dispatches driver to restaurant before food is ready → driver waits → churn
  • 15% cancellation rate — drivers reject dispatches that are too far or wrong direction
  • PostGIS single-primary write ceiling ~500 writes/sec (well below 2K driver-location-updates/sec)
  • No real-time tracking (SMS-only)
Cost: $2K/mo but architecturally broken above 10 cities

Third-party (Uber Direct — Uber Eats's platform-as-a-service)

Pros
  • Uber Direct handles dispatch + driver marketplace + surge + tracking
  • Zero infrastructure to run
  • Sub-30s dispatch guaranteed by Uber
Cons
  • $2-5 per delivery × 30M deliveries/mo = $60M-$150M/yr — kills unit economics
  • Uber sees all your customer + restaurant data (competitive intelligence)
  • Uber owns the customer relationship at drop-off (their driver app, not yours)
  • Strategic dependency on competitor
Cost: $60M-$150M/yr — unit economics destroyed

Cloudflare Workers for edge dispatch

Pros
  • 300+ edge PoPs — sub-30ms globally
  • Edge KV for driver location
  • DDoS-protected at edge
Cons
  • Cloudflare Workers KV eventual consistency (30-60s) unacceptable for dispatch decisions
  • Edge compute cost per invocation × 1M RPS × 30d = $500K+/mo
  • ML inference at edge is expensive ($100K+/mo for GPU inference)
  • Doesn't solve driver marketplace or compliance
Cost: $600K+/mo — 6x more expensive with worse dispatch accuracy

Vertical Postgres + Sidekiq for dispatch

Pros
  • Simpler than Redis + Kafka
  • Team knows Rails stack
Cons
  • Postgres write ceiling well below driver location update rate
  • No real-time tracking
  • Sidekiq FIFO queue doesn't handle dispatch priority
Cost: $5K/mo but ceilinged at ~5 cities
Chosen solution

Redis Cluster + Kafka event stream + ML dispatch (DeepETA-style) + WebSocket tracking + async Sidekiq fulfillment + hybrid driver marketplace

Why
The 3-sided marketplace problem is DIFFERENT from ride-sharing (2-sided). We need to optimize dispatch not just for driver-to-restaurant distance but for: (a) restaurant prep-time prediction — dispatching too early wastes driver time, too late loses fresh food, (b) driver acceptance probability — some drivers reject 40% of dispatches, matching them wastes cycles, (c) real-time surge pricing per hex — customer-facing pricing must reflect actual driver supply, (d) multi-app driver reality — 40% of drivers double-book, we lose them mid-dispatch. ML dispatch handles all four. Redis Cluster + Kafka event stream is the only architecture that supports the write throughput (100K driver-location-updates/sec across all cities at peak) AND the event decoupling (OrderPlaced → RestaurantAccepted → DriverDispatched → PickedUp → Delivered = 5 event stages, each observed by downstream services for notifications/analytics/fraud). Uber Direct is prohibitively expensive at our scale and strategically weak. Cloudflare Workers edge dispatch has 30-60s KV consistency window that breaks dispatch accuracy. The choice is straightforward once you understand the 3-sided marketplace + compound ETA constraints.
Rejected alternatives (with reasons)
  • Naive-nearest on PostGIS — 15% cancellation rate + no prep-time awareness = wrong architecture
  • Uber Direct — $60-$150M/yr destroys unit economics + strategic dependency on competitor
  • Cloudflare Workers edge dispatch — 30-60s KV eventual consistency breaks dispatch accuracy
  • Vertical Postgres + Sidekiq — ceilings at 5 cities; doesn't scale to marketplace
Trade-offs accepted
  • Accept Redis SPOF per shard — plan for shard failover during dinner rush (30-60s of degraded dispatch)
  • Accept ML dispatch GPU infrastructure ($50K+/mo) — pays for itself via 15%→5% cancellation reduction
  • Accept real-time surge pricing PR risk — mitigate with transparent 'demand pricing' UI language
  • Accept 4 stateful surfaces (Kafka + Redis + Postgres + Sidekiq) — dedicated Data Platform SRE team
  • Accept WebSocket infrastructure discipline — reconnect storm handling on deploy is documented runbook
  • Accept multi-app driver reality — dispatch scoring includes 'multi-app penalty' for detected double-booking
  • Accept 20-30 engineer Dispatch Platform team — foundational + high-leverage warrants first-class ownership
Consequences
  • Dispatch accuracy improves 15%→5% cancellation, saves ~$30M/yr in wasted driver dispatches
  • Compound ETA UX becomes trustworthy — customers see 'delivery in 25 min' + it happens
  • Real-time surge pricing generates surge margin ~20% of total revenue during peak periods
  • Kafka event stream becomes reusable primitive for fraud detection, analytics, driver-marketplace incentives, and A/B testing
  • Driver marketplace features (exclusive scheduling, tiered pay, benefits) become strategic moat vs competitors
  • GDPR compliance-as-code discipline flows through every dispatch decision (EU driver classification, data residency)
  • Multi-app-aware scoring becomes competitive advantage — we know which drivers are loyal to us
When would we reverse this decision?
  • If ML dispatch model degrades below simple-nearest baseline for 2+ quarters → simplify to rules-based dispatch + reserve ML for surge pricing only
  • If dispatch platform team drops below 10 engineers → migrate to Uber Direct (accept unit economics hit)
  • If regulatory environment eliminates gig worker classification (all drivers become employees) → rebuild dispatch for W2 scheduling not marketplace
  • If autonomous delivery becomes 60%+ of volume → dispatch architecture shifts from driver-marketplace to route-planning + fleet management

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are engineer #4 at a 2-year-old food delivery startup ($30M ARR, 100 engineers, 15 cities). CTO says Monday: 'Cloud kitchens are 20% of DoorDash's volume by 2028. We need a cloud-kitchen platform. Options: (1) partner with existing cloud kitchen operators (CloudKitchens/REEF/Kitchen United) for revenue share, (2) build our own cloud kitchen software + operate physical kitchens, (3) build software-only cloud kitchen platform (like Shopify for kitchens), or (4) stay out of cloud kitchens entirely and focus on marketplace. Board wants a strategy this Friday. Which do we pick?'

Constraints
  • 1$30M ARR food delivery marketplace, 15 cities, growing 40% YoY
  • 2100 engineers total, 25 on Dispatch Platform team, 10 on Marketplace team
  • 3$50M cash runway (Series C in progress)
  • 4Current unit economics: $2.50 margin per order, 30M orders/mo target
  • 5Cloud kitchens grow 40%/yr — 5% of our volume now, projected 20% in 3 years
  • 6Existing cloud-kitchen partners: 3 (CloudKitchens, REEF, Kitchen United) — small deals
  • 7Regulatory: local health inspection per physical kitchen (state-by-state), commercial kitchen zoning
  • 8Real-estate: cloud kitchen physical footprint = 1000-3000 sq ft + refrigeration + hood ventilation
  • 9Competitive: DoorDash + Uber Eats + Deliveroo all making cloud kitchen investments; Meituan operates 500+ cloud kitchens in China
Your question

What do you recommend Friday? Frame as 3-year plan with unit economics, strategic control, and reversibility. Address the CTO's likely follow-up: 'why not all of them?'

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT
Dispatch cascade failure during Super Bowl Sunday — driver push notification queue backup + ML inference GPU shortage + 400% baseline order volume

PagerDuty alert at 5:47pm ET on Super Bowl Sunday. Order volume at 4.2x baseline (400% higher than a normal Sunday). Dispatch cancellation rate spiked from 5% to 34% in 15 minutes. Customer complaints about 'driver never came' up 800%. Push notification queue lag 45 seconds (baseline 200ms). ML dispatch GPU inference latency 850ms (baseline 30ms). Redis Cluster shard-12 at 98% CPU. Marketing texting: 'we are losing $2M/hour in cancelled orders and customer refunds. Fix this NOW.'

Metrics
  • orders.per_second: 12,400 (baseline 2,900 = 4.2x)
  • dispatch.cancellation_rate: 34 pct (baseline 5 pct = 6.8x)
  • push_notification.queue_depth: 47,000 (baseline 500 = 94x)
  • push_notification.queue_lag_seconds: 45 (baseline 0.2s)
  • ml_dispatch.inference_latency_p99_ms: 850 (baseline 30ms)
  • ml_dispatch.gpu_utilization_percent: 100 (baseline 65 pct — saturated)
  • redis.shard12.cpu_percent: 98 (baseline 40)
  • redis.shard12.mem_percent: 91 (baseline 60)
  • driver.acceptance_rate: 42 pct (baseline 82 pct — drivers rejecting because dispatch info stale)
  • customer.support_tickets_per_second: 34 (baseline 4)
  • estimated_revenue_loss_per_hour: $2.1M
  • restaurants.average_prep_time_delay_min: 18 (baseline 4 = 4.5x — restaurants overwhelmed too)
Logs
  • 17:47:03 dispatch-service: 'ML inference queue depth 8000 (target <100) — scaling up GPU pool from 4 to 8 nodes'
  • 17:48:12 gpu-inference-pool: 'ERROR: cannot allocate GPU node — AWS regional capacity exhausted'
  • 17:48:45 dispatch-service: 'falling back to rules-based dispatch (nearest-driver algorithm)'
  • 17:49:30 push-notification-service: 'FCM API rate-limited (10K/sec cap hit, we are at 12K/sec)'
  • 17:50:14 redis-shard12: 'MEMORY OK 91 pct, evictions starting'
  • 17:51:00 driver-app: '30 pct of drivers report they are receiving stale dispatch info'
  • 17:52:23 customer-support: 'flood of tickets, hold time 25 minutes'
  • 17:53:45 dispatch-service: 'cancellation rate 34 pct, above 20 pct threshold — triggering emergency mode'
  • 17:54:30 marketing-dashboard: 'projected revenue loss $2.1M/hour'
  • 17:55:00 executive-slack: 'Get this fixed. Now.'
Dependency health
  • Redis Cluster: DEGRADED — shard-12 at 98% CPU, other shards healthy
  • ML dispatch: DEGRADED — GPU pool saturated, fell back to rules-based
  • Kafka: HEALTHY — event stream flowing
  • Postgres: HEALTHY at 65% CPU — read replicas handling browse traffic
  • Push notification (FCM): RATE-LIMITED — Google's FCM API 10K/sec cap hit
  • APNS (iOS push): DEGRADED — Apple's API queuing our requests
  • WebSocket gateway: HEALTHY — connections stable
  • AWS: DEGRADED — regional GPU capacity exhausted (Super Bowl affects many services)
  • Restaurants (external): OVERWHELMED — real dependency, we cannot fix from our side
Your investigation
1

It's 5:53pm. Cancellation rate is 34%, revenue loss $2.1M/hour, ML dispatch fell back to rules-based, GPU pool exhausted regionally. What's your SINGLE highest-priority action in the next 5 minutes?

Hint: The bottleneck isn't ML dispatch quality anymore — it's push notification queue lag. Drivers can't respond to dispatches they haven't received yet.
2

Assume push notification queue is now clearing. But ML dispatch is still on rules-based fallback because GPU pool is exhausted. Rules-based dispatch is 15% worse than ML. Should we accept the 15% degradation or try to restore ML dispatch?

Hint: GPU pool is exhausted REGIONALLY. What about cross-region inference? What's the latency + cost trade-off?
3

It's now 6:15pm — 22 minutes after alert fired. Push notifications flowing, ML dispatch running cross-region. But restaurants are STILL overwhelmed — average prep time delay 18 minutes (baseline 4). We can't fix the restaurants from our side. What's your customer-facing action?

Hint: The restaurant bottleneck is real, we can't accelerate cooking. But we can manage customer expectations and prevent further order backlog.
4

Postmortem the next day. Name 3 action items ranked by impact reduction for future 4x-baseline events (Super Bowl, New Year's, Valentine's Day).

Hint: Some are technical, some are capacity planning, some are process. Prioritize by risk-adjusted impact.
5

Draft the customer-facing status page update at T+40 minutes (6:27pm — after push queue cleared + ML dispatch restored, but restaurants still overwhelmed). Constraint: honest, no jargon, sets correct expectations without over-promising fix time.

Hint: Customers experienced cancellations + late deliveries. Some got refunds, some didn't. Manage the narrative honestly.
Knowledge graph

Learn these first

  • 3-sided marketplace dynamics (customer + restaurant + driver)
  • Compound ETA (browse + order + accept + prep + dispatch + delivery)
  • ML dispatch with prep-time prediction (DeepETA-style)
  • Real-time surge pricing per H3 hex + regulatory (gig worker classification)

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.