MASTERCLASS

Gold-standard deep dive

Ticket Booking — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001

Architecture Decision Record

FIFO waiting room + Redis SETNX seat-hold (5-minute TTL) + payment-first confirm + Cloudflare Turnstile bot mitigation (over optimistic-locking-only or database-advisory-locks)

Context

We are building a ticket-sales platform for concerts, sports events, and theater. Peak load is a 'viral drop' — a headline artist releases 60K seats and 3M+ users hit the site in the first 30 seconds. We must prevent double-bookings (two users buying the same seat), prevent bot inventory hoarding (single actors buying thousands of tickets in bot-run scripts), and preserve revenue capture (users who abandon carts free their seat inventory back to the pool). The failure mode is famous: Ticketmaster's Taylor Swift Eras Tour drop in November 2022 was a public disaster — 3.5B requests hit their systems and the resulting outage drew Congressional attention.

Constraints

Peak load: 60K seats, 3M+ concurrent users at drop moment, sustained 200K RPS for the first 5 minutes
Consistency: zero double-bookings — a seat sold to two people is a legal + refund + retention disaster
Fairness: users who arrived first should have a better chance than users who arrived at t+30s
Bot mitigation: single actors must not be able to script-buy 500+ tickets across many accounts
Payment: user must complete payment within 5 minutes of seat hold or the seat returns to the pool
Availability: 99.99% during the drop window (the drop itself is the point; downtime = brand catastrophe)
Latency: p99 seat-hold acquisition < 500 ms; p99 checkout < 2 s including Stripe RTT
Regulatory: some jurisdictions require secondary-market controls (no reselling above face value in some countries)

Options considered

FIFO waiting room + Redis SETNX seat-hold (5-min TTL) + payment-first confirm + Cloudflare Turnstile

Pros

Waiting room is a load-shedding mechanism: only N users at a time can access seat inventory
Redis SETNX gives O(1) atomic seat-lock with automatic TTL expiry — no cleanup job needed
Payment-first confirm means the money is authorized before the seat is committed — no refund overhead
Cloudflare Turnstile is invisible bot mitigation with near-zero user friction (vs. reCAPTCHA which annoys users)
FIFO waiting room enforces arrival-order fairness — the exact anti-scalper primitive
Battle-tested pattern: this is the shape Ticketmaster runs now (post-2022), Fortnite item shop, Nintendo Switch drops

Cons

SETNX TTL expiry is a subtle failure mode — if a payment takes longer than 5 min, the seat can be re-locked by a second user and double-booked
Waiting room requires a fair-queue implementation (Cloudflare's Waiting Room product handles this; DIY is non-trivial)
Turnstile costs money at scale (~$100/mo per 1M challenges — small but real)

Cost: Waiting Room + Turnstile: ~$500/mo baseline. Redis Cluster for seat-holds: ~$3K/mo. Total infra: ~$5K/mo dedicated + Stripe fees.

Optimistic locking on the seat row (SELECT + version check, retry on conflict)

Pros

Zero external dependencies — just Postgres row versioning
Simple mental model: try, fail, retry
Standard textbook pattern

Cons

At 200K RPS during a drop, the retry storm on hot seats generates massive DB load (10-20× amplification)
No natural mechanism to hold a seat for 5 minutes while payment completes — users repeatedly re-verify
Absent a waiting room, ALL 3M users hammer the DB simultaneously — Postgres cannot handle this without extreme sharding

Cost: Compute cost is cheap ($1K/mo Postgres); operational cost of tuning + retry-storm mitigation is high (~2 engineers permanently focused on it).

Database advisory locks (PG's pg_advisory_lock or MySQL's GET_LOCK)

Pros

Database-native — no external state
Automatic release on connection close (safety net)

Cons

Lock contention centralizes on the primary DB — becomes the bottleneck at 200K RPS
No TTL — connection death is the only cleanup mechanism
Advisory locks are non-transactional in some edge cases; correctness is hard to prove

Cost: Similar to option 2 but with worse scaling characteristics.

Buy Cloudflare Waiting Room + Ticketmaster's TM+ Platform (buy)

Pros

Zero engineering — you just embed their JavaScript
Ticketmaster has 20+ years of anti-scalper experience
Regulatory + payment + fraud all handled

Cons

Ticketmaster fees: 5-15% of ticket face value — at $50M annual ticket sales, that's $2.5-7.5M direct fee
Zero differentiation — every customer looks the same as any other Ticketmaster customer
Anti-competitive optics if we are trying to build a Ticketmaster alternative

Cost: $2.5-7.5M/yr at $50M GMV. Punitive at scale.

Chosen solution

FIFO waiting room + Redis SETNX + payment-first confirm + Turnstile (option 1)

Why

Option 1 is the only architecture that structurally handles the 200K-RPS-drop-moment shape. Optimistic locking is a textbook answer that dies at drop-scale (retry-storm math is unforgiving). Advisory locks centralize on the DB primary and cannot survive the load. Buying Ticketmaster kills our differentiation and margin. The chosen shape is what Ticketmaster themselves adopted after the 2022 Taylor Swift disaster — publicly documented. The 5-min SETNX TTL failure mode is real but well-understood: it's the primary cause of the double-booking incident in this masterclass, and the mitigation is documented in the reversal conditions.

Rejected alternatives (with reasons)

Optimistic locking-only — retry storm at 200K RPS is unmanageable and scales badly with # of concurrent users, not # of seats
Advisory locks — DB primary becomes the SPOF; no TTL means orphaned locks require a cleanup job
Buy Ticketmaster's TM+ — 5-15% fee is punitive at scale and eliminates differentiation
Blockchain-based ticketing (NFT-tickets) — solves scalping but adds gas fees + UX friction; not a fit for mainstream events

Trade-offs accepted

Accept the SETNX TTL failure mode (5-min timer race) — mitigate via 'refresh hold' endpoint + strong payment-confirm invariant
Accept the fee on Cloudflare Waiting Room + Turnstile — the alternative (DIY fair queue + captcha) is 6+ months of engineering
Accept that FIFO waiting room means some users wait 30+ min at the drop moment — the alternative (no queue) is a Site-Down public disaster
Accept that Turnstile invisibly fails some legitimate users (~1% false-positive rate) — publish clear escalation path

Consequences

Every ticket product must integrate with the waiting-room service (cannot bypass for special events)
SETNX seat-hold pattern becomes a first-class primitive; every consumer needs 'refresh hold' + 'release hold' semantics
Payment-first confirm requires our checkout to authorize Stripe BEFORE we commit the seat; adds ~600ms latency but eliminates refund overhead
Turnstile challenge failures need a first-class escalation path (customer support flow)
Anti-scalper policy (e.g., 4-ticket limit per account per event) becomes a business rule tracked in our DB, not an engineering guess

When would we reverse this decision?

If Cloudflare Waiting Room pricing changes such that per-drop cost > $10K, evaluate a DIY fair-queue (Redis-backed FIFO with periodic rebalance)
If regulators require server-side proof-of-fairness (some countries do), evaluate an event-sourced audit log of every queue admission decision
If our workload shifts to always-on ticketing (no viral drops), the waiting-room overhead is unnecessary — revert to optimistic locking
If a new anti-bot technology emerges that outperforms Turnstile on false-positive rate, evaluate the swap

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the founding CTO at a live-events startup. Your product currently sells ~500 events/month with low peak load (never more than 5K concurrent users). You just signed a partnership with a major artist for a stadium tour — 50 dates, ~60K seats each. The artist announces the tour drop for 4 weeks from now. Your PM is panicking: 'we need Ticketmaster-quality reliability in 4 weeks or the artist walks and the whole partnership dies.' You have 4 backend engineers. The last time an event of this size dropped (Taylor Swift 2022), the industry leader crashed spectacularly.

Constraints

14-week deadline — the drop is fixed to a public announcement date
24 backend engineers, all senior but none has built ticket-drop infrastructure
350 dates × ~60K seats = 3M tickets total; single-event drop = 60K seats × ~50 concurrent buyers per seat = 3M peak concurrent users at drop
4Availability: 99.99% during the drop window; brand reputation is on the line
5Budget: dedicated ticketing infra for this drop < $50K/mo (4x normal ops)
6Existing stack: Rails + Postgres + Redis + Sidekiq (same as everyone at Series C scale)
7Anti-scalper policy from the artist: 4 tickets max per person per event, verified via ID at door
8The 2022 Taylor Swift precedent is public; the artist team is asking 'what makes you not-Ticketmaster?'

Your question

What architecture do you propose, and how do you communicate the reliability plan to the artist team? Be specific about the waiting-room strategy, the SETNX seat-hold, cost, and the risk-mitigation conversation.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT

Double-booked seat during a viral drop — Redis SETNX TTL race condition

PagerDuty alert at 20:03 UTC during a headline-artist drop. Customer support is on fire — 12 users are reporting they received tickets for seat 24H-Row15-Seat8 (a Section 24H row-15 seat). This is a stadium event; that seat exists exactly once. Payment records show 12 separate Stripe charges, each with our internal payment IDs referencing the same seat_id in our database. The drop opened 3 minutes ago. You are on-call.

Metrics

ticket.confirmed_bookings.rate: normal (~9K/sec during drop peak)
ticket.seat_hold_setnx.rate: 220K/sec (baseline pre-drop 5/sec)
ticket.seat_hold_setnx.hit_rate: 0.28% (correctly rejected because the seat was locked)
ticket.seat_hold_setnx.miss_rate: 99.72% (successfully acquired lock)
ticket.seat_confirmed_race.count: 12 (SHOULD BE 0 — this is the alert trigger)
redis.cluster.p99_latency: 8ms (normal 1-2ms) — cluster is under load but healthy
redis.cluster.evictions_per_sec: 340 (was 0) — SETNX keys near memory limit
payment.stripe.p99: 1.2s (normal 400ms) — Stripe is slower under load
waiting-room.admissions_per_sec: 10K (as configured)
app.checkout.avg_time_between_hold_and_confirm: 4 min 42 sec (was 90 sec baseline) — payments slower under load

Logs

20:00 UTC — drop opens; 3M users hit waiting room; 10K admitted per sec
20:00-20:03 — ~30K users successfully lock a seat via SETNX (5-min TTL)
20:01:15 — user A (in Redis: seat24H-15-8 locked by user-A, TTL 5min)
20:03:00 — user A's payment enters Stripe (slower than baseline due to load)
20:06:15 — user A's SETNX TTL expires (Redis) — seat is unlocked
20:06:16 — user A's payment succeeds; app writes 'seat_confirmed = user-A' to PG
20:06:20 — user B (queued behind user A) attempts SETNX on same seat — SUCCEEDS (because the TTL just expired)
20:06:21 — user B's browser shows 'seat locked'
20:07:30 — user B completes checkout; app writes 'seat_confirmed = user-B' to PG
20:07:31 — PG has BOTH user-A and user-B confirmed on the same seat (no DB-level uniqueness constraint at the check)
20:08:00 — this pattern happens 12 times across the drop
20:03:12 UTC (actual, per PG timestamps) — first support ticket arrives

Traces

Trace of the race for seat 24H-15-8:
User A path:
→ 20:01:15 — SETNX seat24H-15-8 by user-A (TTL 5min)
→ 20:03:00 — begin Stripe checkout (delayed by high load)
→ 20:06:15 — SETNX TTL expires in Redis (user A now has no lock)
→ 20:06:16 — Stripe returns success
→ 20:06:16 — app: INSERT INTO confirmed_bookings (seat_id, user_id) VALUES (24H-15-8, user-A)
User B path (parallel):
→ 20:06:20 — SETNX seat24H-15-8 by user-B — SUCCEEDS (Redis has no lock)
→ 20:06:20 — user B's browser: 'You have this seat! Complete payment in 5 min.'
→ 20:07:30 — Stripe returns success for user B
→ 20:07:30 — app: INSERT INTO confirmed_bookings (seat_id, user_id) VALUES (24H-15-8, user-B)
→ PG has both rows because there's no UNIQUE constraint on (seat_id) — only on (event_id, seat_id, user_id)

Dependency health

Waiting Room: healthy (admitting 10K/sec as configured)
Redis Cluster: healthy but under load; TTLs expiring on time (this is the failure, not a Redis bug)
Postgres: healthy but MISSING the correct uniqueness constraint on confirmed_bookings
Stripe: DEGRADED (2-3× baseline latency) — this is the trigger for the TTL race
App tier: healthy
Customer support: OVERLOADED (12 duplicate-seat tickets in 3 minutes)

Your investigation

You look at the dashboard. Which single fact tells you the most useful thing right now?

Hint: Compare the SETNX TTL to the actual time-from-hold-to-payment.

60 seconds to decide a mitigation. What do you do RIGHT NOW?

Hint: You have two levers: (a) bump the SETNX TTL, or (b) add a uniqueness constraint at the DB. Which is faster?

TTL bump works — no new double-bookings in the next 30 min. What is the postmortem root-cause and top action items?

Hint: The proximate cause was SETNX TTL expiry during slow payment. What's the systemic failure that made it possible for a race to result in a double-confirmed booking?

In the retro, someone says 'we should have known Stripe would be slow.' Would that be a fair postmortem framing?

Hint: The Stripe latency was elevated but Stripe was not down. Whose problem was it that our TTL was calibrated to baseline?

The artist team asks: 'should this have been caught in the load test we ran 2 weeks ago?' What's the honest answer?

Hint: What did the load test cover, and what would it need to cover to catch this?

Knowledge graph

Learn these first

Redis SETNX with TTL semantics + fencing tokens (Kleppmann's redlock analysis)
FIFO fair-queue admission control (Cloudflare Waiting Room)
Bot mitigation basics (Turnstile / reCAPTCHA / device fingerprinting)
Timeout calibration under load (p99 vs. baseline p50)

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.