Skip to main content
Back to E-commerce
MASTERCLASS
Gold-standard deep dive

E-commerce — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001
Architecture Decision Record

Redis-based inventory reservations (SETNX with TTL) + optimistic locking on final commit + eventual DB reconciliation (over pure pessimistic DB locks, pure optimistic locks with retry, or CQRS with event sourcing)

Context
We are building an ecommerce marketplace at $10B+ GMV scale (Shopify BFCM 2024: $11.5B GMV in 4 days; Amazon Prime Day 2024: sustained 6,311 Aurora writes/sec). Peak load is BFCM: 500K+ concurrent shoppers, 50K orders/min, 10× normal traffic. Inventory oversell (selling more units than we have) is catastrophic — legal risk + customer trust damage + refund cost. Undersell (rejecting valid orders due to false-positive lock) is a competitive disadvantage. We have millions of SKUs; hot SKUs during flash sales (limited-drop, celebrity endorsement) see 100K+ concurrent buyers.
Constraints
  • $10B+ annual GMV; peak 50K orders/min during BFCM
  • 500K+ concurrent shoppers during peak
  • Availability: 99.99% checkout tier; oversell is a hard failure (cannot ship what we don't have)
  • Latency: p99 add-to-cart < 200 ms; p99 checkout < 3 s (including Stripe RTT)
  • SKU count: millions; hot SKUs 100K+ concurrent buyers during drops
  • Regulatory: consumer protection laws (misleading availability = fine); PCI-DSS for payment
  • Multi-region: US + EU + APAC; different regional inventory pools
  • Team: 100 engineers on ecommerce; 20 on the inventory + cart subsystem
Options considered

Redis SETNX + TTL for holds + optimistic locking on final commit + async DB reconciliation (chosen)

Pros
  • Fast holds — SETNX is O(1); can serve 100K holds/sec on a single Redis shard
  • TTL provides automatic cleanup for abandoned carts (no manual sweep job)
  • Optimistic commit ensures no oversell — the CAS on final commit is the correctness guarantee
  • Redis is the industry-standard inventory-reservation primitive (Shopify, Etsy, Amazon all use variants)
  • Async DB reconciliation lets us handle bursts without DB write-throughput becoming the bottleneck
Cons
  • SETNX TTL race is a real failure mode (see masterclass incident) — the seat-booking version applies here too
  • Requires calibrating TTL to p99 checkout latency (not p50)
  • Redis is a hard dependency — Redis outage = no new checkouts (mitigated by Redis Cluster + multi-region)
  • Reconciliation between Redis + DB must be tight — drift causes phantom oversell
Cost: Redis Cluster + reconciliation service ~$50K/mo at Shopify scale. DB write savings dwarf this.

Pessimistic locks in DB (SELECT ... FOR UPDATE)

Pros
  • Simple + correct at low concurrency
  • No cache-DB drift
Cons
  • SELECT FOR UPDATE holds DB row lock across the entire HTTP round-trip (including client + payment provider round-trips)
  • At BFCM concurrency, DB row locks become the bottleneck; timeouts + deadlocks proliferate
  • Cannot serve 50K orders/min through DB row locks without extreme sharding
Cost: Low upfront but breaks catastrophically at drop-scale.

Optimistic locking DB-only (INSERT ... WHERE version = $, retry on conflict)

Pros
  • No lock contention on read path
  • Correctness via version-check on commit
Cons
  • During flash sales, retry storms hammer DB (10-100× amplification for hot SKUs)
  • Without a cache tier, DB read throughput becomes the constraint on cart-add rate
  • No 'reservation' semantics — users see 'in stock' then get 'sold out' at checkout
Cost: Compute-heavy at scale; UX degrades under load.

CQRS with event-sourced inventory ledger

Pros
  • Perfect audit trail — every inventory change is an event
  • Time-travel debugging
Cons
  • Complexity — CQRS + event sourcing has a steep learning curve
  • Materialized views take time to converge; 'is this in stock' becomes eventually-consistent
  • Rebuild time after schema change is O(events) — hours to days at scale
Cost: Engineering complexity + eventual-consistency UX cost.
Chosen solution

Redis holds + optimistic commit + async reconciliation (option 1)

Why
Ecommerce at BFCM scale demands three properties simultaneously: (a) fast in-memory reservation semantics so shoppers see accurate 'in stock' without hitting DB, (b) correctness guarantees so we never oversell, (c) automatic cleanup for abandoned carts. The chosen architecture achieves all three. Pessimistic DB locks fail at concurrency. Optimistic-only kills UX (in-stock → sold-out at checkout). Event-sourcing is over-engineered for the read path (users want to see 'in stock,' not the entire inventory event history). Every major ecommerce platform (Shopify, Etsy, Amazon, eBay) uses variants of this Redis + DB pattern. It's proven to work.
Rejected alternatives (with reasons)
  • Pessimistic DB locks — bottleneck at drop-scale
  • Optimistic-only — UX degradation during flash sales
  • CQRS + event sourcing — over-engineered; eventual-consistency UX issue
  • Skip reservations (best-effort inventory) — leads to oversell + lawsuits
Trade-offs accepted
  • Accept the Redis SETNX TTL race — mitigate via TTL calibration + fenced tokens on commit
  • Accept the async DB reconciliation lag — mitigate via bounded lag + safety inventory buffer
  • Accept Redis as critical dependency — mitigate via Redis Cluster + multi-region
  • Accept the operational discipline of 'monitor Redis-DB drift' — dashboard + alert + auto-reconcile
Consequences
  • Inventory service becomes a first-class discipline (dedicated team owning Redis + reconciliation)
  • Every SKU has a per-region Redis reservation namespace
  • Cart TTL becomes a business decision (5 min? 15 min? 30 min?)
  • Flash-sale mode has different TTL + reservation semantics than normal browsing
  • Reconciliation jobs run continuously; drift alerts page on-call
When would we reverse this decision?
  • If our workload shifts to always-on scarcity (no flash sales), simpler locking may suffice
  • If Redis pricing changes dramatically, evaluate alternatives (Aerospike, DynamoDB DAX)
  • If we adopt a full-event-driven architecture platform-wide, event-sourcing may become worth the complexity
  • If our SKUs move to digital-goods-only (no physical inventory), the whole reservation problem disappears

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the founding CTO at a direct-to-consumer (DTC) apparel brand. You've built a following on Instagram and want to launch your own ecommerce site for BFCM (12 weeks away). Your CEO says 'we need Amazon-quality checkout with Prime-Day-like flash sales.' Your product team is planning a limited-edition drop for Black Friday morning (500 units, expected 50K interested shoppers hitting the site in the first 5 minutes). You have 4 engineers, no dedicated ecommerce experience, and $20K/mo infrastructure budget.

Constraints
  • 112-week deadline (BFCM is a fixed date)
  • 24 backend engineers; no ecommerce infrastructure experience
  • 3Flash sale: 500 units, 50K expected concurrent shoppers in first 5 min
  • 4Ongoing traffic: 5K daily orders (steady state)
  • 5Existing stack: Vercel + Postgres + Redis on AWS
  • 6Budget: dedicated ecommerce infra < $20K/mo
  • 7Payment: Stripe (already integrated)
  • 8CEO's ask: 'Amazon-quality with Prime-Day flash sales'
Your question

What architecture do you propose, and how do you scope 'Amazon-quality checkout' into 12 weeks with 4 engineers? Be specific about buy vs. build, flash-sale handling, and the CEO conversation.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT
Flash-sale oversell — 500 units advertised, 3200 orders placed

PagerDuty alert at 09:03 EST on Black Friday morning. Our limited-drop went live at 09:00 EST (500 units). Alert: 'inventory reconciliation shows orders_placed=3200 but items_sold_advertised=500 for SKU LTD-2024-BLK-M.' Our fulfillment team is looking at 3,200 charged orders for 500 units. Customer support Slack: EXPLODING. You are on-call.

Metrics
  • orders.placed_last_5min.SKU_LTD-2024-BLK-M: 3200
  • inventory.units_available.SKU_LTD-2024-BLK-M: 500 (advertised)
  • inventory.oversell_ratio: 6.4× (charged 3200 for 500 units)
  • redis.setnx_operations_last_5min: 50,000 (all 50K attempted)
  • redis.setnx_success: 500 (correct — only 500 succeeded)
  • db.orders_committed: 3200 (WRONG — should be at most 500)
  • stripe.charges_captured: 3200 × $89 = $284,800 (all captured)
  • reconciliation.redis_db_drift: 2700 (Redis 500 correctly locked; DB says 3200 orders)
  • recent_deployment: order-service updated 2 hours ago (bug fix: 'Handle Redis timeout gracefully')
Logs
  • 09:00:00 EST — drop goes live; Cloudflare Waiting Room admits 10K users
  • 09:00:15 — first orders arrive; Redis SETNX correctly acquires locks for first 500 users
  • 09:00:16 — SETNX returns 0 (already locked) for subsequent users; they see 'sold out'
  • 09:00:45 — one Redis node briefly overloaded (50K SETNX requests hit)
  • 09:00:47 — Redis timeouts begin (~2% of requests)
  • 09:00:47 — the order-service (deployed 2 hours ago) 'handles Redis timeout gracefully' by...
  • 09:00:47 — ...proceeding to allow the order without a Redis reservation ("soft fail — don\'t block on Redis")
  • 09:00:47-09:03:00 — 2,700 orders committed without valid Redis reservations
  • 09:03:00 — reconciliation job runs; detects Redis-DB drift; alert fires
Traces
  • Trace of an oversell order:
  • → User in Waiting Room admitted at 09:00:47
  • → Add to cart → order-service.reserve_inventory('LTD-2024-BLK-M')
  • → Redis SETNX: TIMEOUT (2 second timeout hit due to node overload)
  • → order-service: recent deploy says 'if Redis times out, log warning + proceed'
  • → order-service: INSERT order into PG (bypassing reservation check)
  • → Stripe: CAPTURE payment (bypassing reservation check)
  • → User sees confirmation page → 'Order placed! Delivery in 3-5 days'
  • → INSERT commits to PG despite no valid Redis lock
  • The recent deploy INTRODUCED this bug in the name of 'graceful Redis failure.'
Dependency health
  • Cloudflare Waiting Room: healthy
  • Cloudflare Turnstile: healthy (bot mitigation worked)
  • Redis: DEGRADED (single node overloaded; 2% timeout rate)
  • Postgres: healthy (accepting the wrongful INSERT operations)
  • Stripe: healthy (charging all cards including oversell)
  • Customer trust: DEGRADED
Your investigation
1

What is the ROOT cause?

Hint: The recent deploy. What did it change?
2

60 seconds to decide a mitigation. What do you do RIGHT NOW?

Hint: You need to (a) stop new oversells + (b) handle the 2,700 wrongful orders. Which first?
3

Rollback stops new oversells. Now: 2,700 users have been charged for 500 units. What's your cleanup + communication strategy?

Hint: Decision: refund all, fulfill first-500 + refund rest, or randomize?
4

Postmortem: what should we change so this NEVER happens again?

Hint: The bug was a 'graceful fail-open.' What's the systemic fix?
5

The engineer who wrote the fail-open code is understandably worried. What's the retro tone?

Hint: Blameless-postmortem discipline.
Knowledge graph

Learn these first

  • Redis SETNX + TTL semantics (calibrated to p99, not p50)
  • Optimistic locking + CAS on final commit
  • Fail-closed vs. fail-open discipline for correctness-critical paths
  • Cloudflare Waiting Room + Turnstile for flash-sale UX

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.