MASTERCLASS

Gold-standard deep dive

Ride Sharing — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001

Architecture Decision Record

DISCO 30-second bipartite batched matching over sync per-request or marketplace-auction dispatch at metros-scale (~1M rides/day, single-country)

Context

We are a mid-scale ride-hailing platform operating 12 cities across one country with ~1M rides/day. Our current dispatch is sync per-request with a PostGIS + Redis geo-set stack (the L5 architecture). During peak hours, rider p99 wait-time-to-offer has crept from 3 s to 11 s as driver density thins in specific H3 cells. Driver utilization is at 32% (industry average is 55%+). Two competing proposals are on the table for the next 6-month engineering roadmap: (a) build a DISCO-style 30-second bipartite batch matcher, (b) launch a marketplace-auction dispatch (drivers bid on rides). We must decide before Q1 planning locks.

Constraints

Team size: 12 backend engineers across 3 pods (dispatch, driver-experience, rider-experience)
Timeline: 6-month build window; want first improvements shipped in month 2 for on-going KPI defense
Budget: total dispatch tier infrastructure < $500K/mo (currently $180K/mo)
Availability: 99.99% globally, matches current
Driver-utilization improvement target: 32% → 45% within 12 months
Rider-latency target: p99 offer within 15 s (currently 11 s median)
Regulatory: driver-earnings floor policy in half our cities forbids some marketplace-auction mechanics
Legacy: existing PostGIS + Redis geo-sets are working and stable; can extend but not replace in 6 months

Options considered

DISCO 30-second batched bipartite matching (Uber-style)

Pros

Improves driver utilization by ~15-20 points empirically (Uber has published this)
Enables surge pricing as a first-class output of the batch
H3-cell partitioning trades ~5% global optimality for a ~50× speedup — mathematically well-understood
Kuhn-Munkres bipartite is a mature O(n³) algorithm; production implementations exist
Compatible with our existing Redis geo-set state without a rip-out

Cons

30-second UX tax is real — some riders will bounce and use a competitor
First ship is ~3 months (algorithm) + 2 months (Kafka topology + Ringpop sharding) + 1 month (integration)
Requires ~4 engineers full-time; other pods lose that headcount

Cost: Engineering: ~24 engineer-months. Infra: +$70K/mo Kafka + DISCO fleet. Total year-1: ~$2.4M.

Marketplace auction dispatch (drivers bid on rides)

Pros

Aligns driver incentives explicitly — the top-bidder wins
Enables dynamic personal pricing (a driver who prefers longer rides can bid down on short ones)
Simpler algorithmically — no bipartite matching, just an auction market clearing every N seconds

Cons

Regulatory: 6 of our 12 cities have driver-earnings-floor policies that forbid explicit bidding
Predatory driver behavior: high-tier drivers over-bidding on high-margin rides, starving low-tier drivers
Race conditions: what if two drivers accept the same offer? Requires distributed locking (Zookeeper/etcd)
No existing production ride-hailing platform runs marketplace-auction at scale — this is greenfield

Cost: Engineering: ~30 engineer-months (regulatory work alone is 6 months). Infra: unclear. Total year-1: ~$3M+, regulatory-risk-adjusted.

Micro-batched dispatch (5-second batches, PostGIS + Redis, extend existing)

Pros

Ship in ~2 months — no new infra, no algorithm work
Improves utilization by ~5-8 points (published estimate from Lyft's approach)
Reversible: if it doesn't help enough, DISCO is still the answer next quarter
No UX tax beyond 5 s — riders barely notice

Cons

Ceiling: micro-batching cannot reach DISCO's 15-20 point utilization gain
Some structural KPIs (driver-fair-earnings, cross-cell arbitrage) still not addressed
Might reach a wall at 1M+ rides/day where sync-per-request even in micro-batches falls behind

Cost: Engineering: ~4 engineer-months. Infra: negligible. Total year-1: ~$200K.

Google Directions API + sync matching + no in-house dispatch algorithm

Pros

Zero engineering effort on dispatch — the API handles everything
Fully managed

Cons

Directions API is for routing, not dispatch — it doesn't solve the bipartite matching problem
$0.005 per request × 1M/day × 30 = $150K/mo, and it climbs linearly with scale
No control over surge pricing, utilization, or driver-experience decisions
Would be a strategic dependency on a competitor (Google Maps team owns their own ride-hailing product too)

Cost: $1.8M+/yr just in API fees at current scale. And no differentiation.

Chosen solution

DISCO 30-second batched bipartite matching

Why

DISCO is the only option that structurally improves the KPI we set (driver utilization 32% → 45%). Micro-batching gets us halfway but hits a ceiling. Marketplace auction is a regulatory dead-end in half our markets. Google Directions is a non-answer — it doesn't solve the matching problem. The 30-second rider UX tax is real but manageable: our rider retention data shows the p50 rider currently waits 6 s regardless of matching strategy, so the additional 24 s of 'searching...' UX comes with a competitive-latency-story trade we can absorb. Ship micro-batching first (month 2) as a bridge, then swap to DISCO in month 6. Reversal is intact — if DISCO doesn't hit the KPI in a year, we revisit.

Rejected alternatives (with reasons)

Marketplace auction — 6 of 12 cities have earnings-floor policies that forbid the mechanic
Google Directions API — doesn't solve the matching problem, and creates a strategic dependency on a competitor
Do nothing — utilization is already below industry benchmark and driver churn correlates with it

Trade-offs accepted

Accept 30-second rider UX tax as the cost of ~15-point utilization improvement
Accept ~5% global optimality loss due to H3-cell partitioning (Kuhn-Munkres per cell rather than global)
Accept +$70K/mo infrastructure cost during ramp-up
Accept 6-month time-to-full-value; ship the micro-batched bridge in month 2

Consequences

Driver utilization improves to ~40-45% within 12 months of DISCO ship
Surge pricing becomes a first-class output of the batch — enables previously-impossible dynamic-pricing UX
Rider p99 wait-to-offer goes from 11 s → 30-35 s; measured impact on cancellation rate needs to stay below 3%
The 4-engineer dispatch pod becomes a permanent org unit — DISCO is not maintainable as a side-project
Kafka + Ringpop expertise becomes a hiring priority for the pod

When would we reverse this decision?

Driver utilization does not exceed 38% at the 12-month mark → revisit; possibly add ML retention model earlier
Rider cancellation rate rises above 5% due to the 30-s tax → shorten batches to 20 s (at ~10% utilization loss) or add L1 sync-fast-path for premium tiers
Regulatory change makes marketplace-auction viable in majority of cities → evaluate hybrid: DISCO base + auction overlay for driver preferences
Cross-region expansion doubles operational overhead → evaluate a managed dispatch platform (does not currently exist, but a startup could emerge)

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the senior backend lead at a Series C food-delivery startup that just closed $80M in funding. Your CTO wants to add a ride-hailing product for a specific market segment (medical transport for elderly patients). She has read Uber's engineering blog and told the board 'we'll ship Uber-style dispatch in 12 weeks.' You have 3 backend engineers, and none of you has built a real-time bipartite matcher. Your launch market is one mid-sized city (~300K population). The medical-transport niche means ~5K rides/day realistic peak, all pre-booked at least 15 minutes ahead.

Constraints

112-week deadline (board commitment for a specific fundraise milestone)
23 backend engineers, none with distributed bipartite matching experience
3One launch city, ~300K population, ~5K rides/day realistic peak
4All rides are pre-booked ≥15 minutes ahead (medical appointments)
5Availability: 99.9% during business hours; overnight can be lower
6Budget: total dispatch tier < $30K/mo (small niche market economics)
7Regulatory: medical transport = licensed driver pool, ~200 total
8The CTO's 'Uber-style dispatch' remark is aspirational — she wants the KPI outcomes (high utilization, low rider wait), not the specific architecture

Your question

What architecture do you propose, and how do you push back on the 'Uber-style dispatch' framing? Be specific about product choice, deployment shape, cost, timeline, and the pushback conversation.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT

Surge pricing collapse on New Year's Eve — drivers went home, riders got 8× fares

PagerDuty alert at 23:47 on Dec 31st. Surge multipliers in 4 major H3 cells (all downtown clubs/venues) have spiked to 8.2× within 15 minutes. Normal peak surge for New Year's Eve is 2.5-3.2×. Rider cancellation rate on those cells has spiked to 71%. Simultaneously, driver-online count in those cells has dropped 40% — drivers are going home rather than accepting rides. Twitter is on fire. You are on-call.

Metrics

surge.cell_A.multiplier: 8.2× (was 2.4× 15 min ago) — cell = Downtown club district
surge.cell_B.multiplier: 7.9× (was 2.1×) — cell = Old Town venue district
surge.cell_C.multiplier: 8.5× (was 2.6×) — cell = Waterfront district
surge.cell_D.multiplier: 8.1× (was 2.3×) — cell = Airport
rider.cancellation_rate: cells A/B/C/D at 71% (baseline for NYE: 12%)
driver.online_count: cells A/B/C/D at 60% of 30-min prior (drivers are going home)
rider.request_rate: cells A/B/C/D at 220% of 30-min prior (post-midnight surge)
kafka.consumer_lag.surge_pricer: 47 s (was 0.3 s) — Flink is behind
flink.watermark_lag: 44 s (should be <2 s)
flink.job.exception_rate: 0.2/s (baseline 0)

Logs

23:00-23:44 — normal NYE ramp, surge climbing 2.0× → 2.5×
23:44 — Flink job 'SurgePricer' logged: 'clock skew detected on task manager 3'
23:45 — Flink watermark lag begins climbing
23:46 — surge computation window mis-aligned; supply/demand delta computed against wrong 15-s window
23:47 — output surge multipliers jumped by 3-4× in affected cells
23:47 — PagerDuty alert fires
23:48 — driver app UI shows 8× surge; some drivers accept; most view it as 'system glitch' and log off
23:52 — rider app UI shows 8× surge; cancellation rate spikes

Traces

Trace of a surge computation for cell A at 23:46:
→ Flink SurgePricer job (task manager 3)
→ Window: 15-second tumbling window ending at 23:46:00
→ Expected: aggregate GPS-ping count for cell A between 23:45:45 and 23:46:00
→ Actual: aggregated between 23:45:00 and 23:46:00 (60 seconds due to clock skew)
→ Supply signal: 4× normal (because the window was 4× wider)
→ Demand signal: correctly windowed via a different task manager
→ Ratio: 8× normal (supply denominator was under-counted)
→ Output surge multiplier: 8.2×
Contrast: cell E (not affected by task-manager-3 skew) — surge is a healthy 2.7×.

Dependency health

Regional ingest: healthy
Kafka (gps-pings, ride-requests topics): healthy, ISR full
Flink SurgePricer job: DEGRADED (task manager 3 clock skew)
Flink task managers 1, 2, 4, 5: healthy
Redis surge state: healthy (receiving the bad values)
DISCO: healthy (accepting the bad surge as input)
Rider app: healthy but showing 8× UI
Driver app: healthy but seeing 8× and behaving accordingly

Your investigation

You look at the dashboard. Which single metric tells you the most useful thing right now?

Hint: The customer-visible symptom is surge at 8×. What's the upstream cause?

You correlate the Flink watermark lag with the 'clock skew detected on task manager 3' log line. What's your hypothesis for why exactly some cells are affected?

Hint: Flink processes different Kafka partitions on different task managers. What determines which partition serves a given H3 cell?

60 seconds to decide a mitigation. What do you do RIGHT NOW?

Hint: You have two levers: (a) fix the Flink task manager, or (b) freeze surge output to a safe fallback. Which is faster to execute at 23:52 on New Year's Eve?

Feature-flag freeze works — surge drops to 2.5× within 40 seconds. Driver count recovers to 90% within 5 min, cancellation rate recovers to 15% within 10 min. What is the postmortem root-cause and top action item?

Hint: The proximate cause was the clock skew on TM3. What's the systemic failure it exposed?

In the retro, someone asks 'should we have caught this on the December 31 Prior New Year's Eve tests?' Would we have? Why or why not?

Hint: What did the load-testing look like? What's unique about a leap-second / clock-skew scenario?

Knowledge graph

Learn these first

Bipartite matching (Kuhn-Munkres / Hungarian algorithm)
Kafka partitioning + Flink windowed streaming
H3 hexagonal geospatial indexing (https://h3geo.org/)
Consistent hashing at the coordinator layer (Ringpop-style)

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.