MASTERCLASS

Gold-standard deep dive

Payment System — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001

Architecture Decision Record

Double-entry ledger + IETF Idempotency-Key + Saga orchestration for cross-service consistency (over event-sourcing or synchronous 2-phase commit)

Context

We are building an in-house payment platform for a marketplace product. Payments must be strictly durable, exactly-once with respect to the merchant view, and support refunds, chargebacks, disputes, and holds. Volume is currently $100M annual GMV, growing 40% YoY. Regulatory: PCI-DSS Level 2 minimum, EU-PSD2 SCA in 6 countries. We already have a modern service architecture (order-service, inventory-service, notification-service). The open question is the ledger + consistency + orchestration design.

Constraints

Consistency: money in a merchant account MUST equal sum-of-signed-transactions. Zero drift.
Idempotency: client retries MUST NOT double-charge. IETF draft spec is the reference (`Idempotency-Key` HTTP header).
Regulatory: PCI-DSS Level 2, EU-PSD2 SCA, US FinCEN reporting; auditor-visible append-only history required
Throughput: 100 TPS steady-state, 5,000 TPS peak (Black Friday); grow to 50,000 TPS in 3 years
Latency: p99 authorize < 800 ms end-to-end (includes external card network round-trip)
Refund/chargeback flows must reference the original charge deterministically
Cross-service atomicity: charge involves order-service (reserve inventory), payment-gateway (call Stripe/Adyen), notification-service (email receipt) — all must succeed or the whole flow rolls back

Options considered

Double-entry ledger + IETF Idempotency-Key + Saga orchestration (with compensating actions)

Pros

Double-entry is the auditor-familiar bookkeeping standard — regulators are trained on it
IETF Idempotency-Key spec is well-supported by Stripe, Adyen, Braintree, PayPal — reuse client libraries
Saga is the industry-standard pattern for multi-service consistency without 2PC (Netflix, Uber, AWS all use variants)
Compensating actions are auditable — every refund/reversal is a first-class transaction, not a delete
Scales to 50K+ TPS by partitioning the ledger by merchant_id (each merchant is a shard)

Cons

Saga is complex — you must design every compensating action explicitly
Double-entry with hot-key merchants (major sellers) requires per-merchant queue serialization
Idempotency-Key deduplication requires a durable, low-latency store (Redis-backed, but PG-durable for audit)

Cost: Engineering: ~9 engineer-months. Ops: adds ~$1.5K/mo for ledger DB + idempotency cache. Total year-1: $650K including team time.

Event-sourced ledger (all state derived from a Kafka event log)

Pros

Every state change is an append-only event — perfect for audit + replay
Time-travel debugging is trivial
Any read view (per-merchant balance, per-day GMV, per-dispute status) is a materialized projection

Cons

Regulators do not know event sourcing — every audit becomes a training exercise
Materialized-view rebuild after a schema change is O(all events) — hours to days at scale
Debugging is hard: 'what was the state at 3pm Thursday?' requires replaying events, not looking at a snapshot
Adoption cost is high — the whole team must learn CQRS + eventual-consistency semantics

Cost: Engineering: ~14 engineer-months (including team ramp-up). Ops: ~$4K/mo for Kafka + view stores. Total year-1: $900K+.

Synchronous 2-phase commit across services (order + payment + inventory)

Pros

Simple mental model — either everything succeeds or nothing does
No compensating actions needed

Cons

2PC is a liveness disaster: any coordinator crash leaves participants blocked
Requires distributed locks; latency floor is 200-500 ms per commit
Every service becomes a hard dependency of every other — cascading failures during peak
Industry consensus: 2PC does not scale past ~1K TPS in a real multi-service topology

Cost: Engineering: ~4 engineer-months (fastest). Ops: hidden cost in outages. Total year-1: unpredictable due to outage risk.

Buy Stripe Connect (managed marketplace payments)

Pros

Zero regulatory work — Stripe is PCI-DSS Level 1 and handles PSD2 SCA
Ships in ~4 weeks (integrate their SDK + webhooks)
No custom ledger — Stripe operates the ledger for you
Massive team-time savings — 3 engineers ship, not 8

Cons

Stripe Connect fee: 0.25% + $2 per transaction — at $100M annual GMV, that's $500K/yr direct fee
Vendor lock-in on a critical path; migration off Stripe is a 6+ month project
Payout timing controlled by Stripe (2-day standard, faster only at premium fee)
Custom marketplace features (holds, escrow, split payouts) are Stripe-flavored, not fully custom

Cost: Engineering: ~1.5 engineer-months. Ops fee: $500K/yr (0.5% of GMV). Total year-1: ~$580K.

Chosen solution

Double-entry ledger + IETF Idempotency-Key + Saga orchestration (option 1)

Why

At $100M GMV growing to $500M+ in 3 years, Stripe Connect's variable fee becomes the dominant cost ($500K → $2.5M+/yr). At that trajectory, building our own is a positive-NPV investment even at $900K year-1 cost. Event-sourcing is a legitimate alternative but adds team-ramp-up cost and regulator-friction that we can avoid. 2PC is a non-starter at any scale. The chosen option preserves auditor-friendliness (double-entry is universal), reuses the IETF spec that our client libraries already implement (Stripe SDK-style Idempotency-Key handling), and scales via merchant-sharding without needing a custom coordinator. This is the Uber / Stripe / Adyen shape and there is public engineering literature for every piece.

Rejected alternatives (with reasons)

Event sourcing — adds team-ramp-up + regulator training + rebuild cost; the auditability gain is real but not worth the friction at our stage
2PC — proven not to scale in a multi-service topology; cascading-failure risk unacceptable for payments
Stripe Connect (buy) — legitimate MVP answer, but the variable fee at 3-year projected GMV makes it more expensive than build
SQL Serializable transactions across services — same theoretical problem as 2PC; hides the coordination but doesn't remove it

Trade-offs accepted

Accept the complexity of Saga design (every step needs a compensating action)
Accept per-merchant queue serialization for hot merchants (major sellers) — the throughput ceiling is per-merchant, not global
Accept the storage cost of an append-only ledger — no updates, no deletes, ever (regulator requirement)
Accept that Idempotency-Key requires a low-latency + durable store — Redis-backed with PG-durable audit

Consequences

Every service that touches payments must adopt Saga semantics (order-service, inventory-service, notification-service)
Every payment endpoint must accept Idempotency-Key header; missing key becomes a client bug
Auditors get a clean double-entry ledger — regulator conversations are shorter
Merchant-sharding creates a clear scale story: 'we can add capacity by adding shards; hot merchants are their own scale problem'
New engineers on the payments team must learn Saga + double-entry as pre-reqs (2-week ramp)

When would we reverse this decision?

If our team churn causes Saga-complexity to become the top outage cause, revisit — event sourcing has simpler semantics, and the ramp cost is one-time
If a regulator explicitly requires event-sourcing evidence chains (some crypto/CBDC contexts do), event-sourcing becomes non-optional
If our GMV shrinks such that Stripe Connect's fee is now cheaper than in-house ops, migrate back (reversible over ~2 quarters)
If a new managed service emerges (Stripe Terminal / Adyen Marketplace) with feature parity + lower fees, revisit the buy-vs-build calculus

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the tech lead at a Series B marketplace startup (think early Etsy or StockX). Your product currently uses Stripe Connect for all payments — this has worked fine at $10M annual GMV, but you're now at $150M projected and the Stripe fee ($750K+/yr) is now visible on the board deck. The CFO asks: 'why are we paying Stripe $750K when we could just build our own payments team?' Your CEO wants a proposal in 2 weeks. You have 4 backend engineers, and none of them has built a payments system before.

Constraints

1Current: $150M projected GMV, using Stripe Connect at ~0.5% marketplace fee = $750K/yr
2Growth: 40% YoY projection → $210M next year, $290M year after
3Team: 4 backend engineers, all senior but none with payments experience
4Regulatory: 12 countries in operation (US + EU + UK + Australia); PCI-DSS Level 2 minimum
5Deadline: proposal in 2 weeks; ship decision in 8 weeks; execution over 12-18 months if we go build
6Board is sensitive to reliability regressions — a single payment outage during Q4 would tank retention
7CFO's framing 'we could just build our own payments team' understates the scope by ~10× (Stripe has 200+ engineers on payments)

Your question

What do you propose to the CFO and CEO, and how do you frame the buy-vs-build decision honestly? Be specific about costs, timelines, risks, and the phased plan.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT

Double-charge incident: split-brain across regions during a cross-region network partition

PagerDuty alert at 10:23 UTC on a Monday. Merchant support tickets are pouring in — customers report being charged twice for the same order over the last hour. Payment-service dashboard shows nothing anomalous. The Idempotency-Key store shows single entries per charge. But CFO's on-call query 'SELECT COUNT(*) FROM charges WHERE created_at > NOW() - INTERVAL 1 HOUR' shows 12% more charges than expected. You are on-call.

Metrics

payment.charges.rate: 118% of baseline (was 100% at 09:23 UTC)
payment.idempotency_lookups.rate: 100% of baseline (normal)
payment.idempotency_hit_rate: 91% (was 91% — normal)
payment.charges.duplicate_by_orderid: 12% (was ~0.03% — HUGE spike)
merchant.support.tickets: 340 in last hour (baseline: ~15)
region.us-east-1.replication_lag_to_us-west-2: 47 s (was <1 s)
region.us-east-1.idempotency_cache_writes: normal
region.us-west-2.idempotency_cache_writes: normal
cross-region-network.tcp-retransmits: 4.2%/sec (was 0.01%/sec) — network is degraded but not down
load-balancer.geo-routing.mismatched-region-percentage: 8% (baseline <0.1%)

Logs

09:23 UTC — normal operations
09:47 UTC — cross-region network provider (fiber-cut in Chicago) announces partial packet loss between us-east-1 and us-west-2 datacenters
09:48 UTC — internal load balancer starts geo-routing based on primary DNS, but some clients have cached DNS from either region
09:49 UTC — merchant clients start seeing 500 errors on their status-poll (they poll charge status after submit)
09:50 UTC — merchant client-side retry logic kicks in — retry with the same idempotency key
09:50 UTC — retry hits the OTHER region's payment-service (DNS routing landed differently)
09:50 UTC — other region's Idempotency cache does NOT have this key (cross-region replication lag is 47s)
09:51 UTC — payment-service in other region processes it as a NEW charge
09:51 UTC — original region's payment-service also processed it — now TWO charges exist for same idempotency key
10:00 UTC — cross-region replication catches up, both regions now have duplicate entries in the ledger
10:15 UTC — merchant support tickets start climbing
10:23 UTC — PagerDuty alert fires on payment.charges.duplicate_by_orderid

Traces

Trace of merchant's duplicate charge:
→ Client (POST /v1/charges with Idempotency-Key: abc123) → LB → us-east-1 payment-service
→ us-east-1 checks Redis idempotency cache → MISS (first time)
→ us-east-1 processes charge → calls Stripe/Adyen → gets auth OK
→ us-east-1 writes to ledger + writes idempotency key to cache with 24h TTL
→ us-east-1 returns 201 to LB → LB fails to return to client (network hiccup at 09:49)
→ Client sees timeout → retries with SAME Idempotency-Key: abc123
→ This time DNS resolves to us-west-2 (client's DNS cache expired during the network blip)
→ us-west-2 checks Redis idempotency cache → MISS (cross-region replication is 47s behind)
→ us-west-2 processes charge → calls Stripe/Adyen → gets auth OK for the SECOND time
→ us-west-2 writes to ledger — DUPLICATE
Contrast: had cross-region replication been sync (or the cross-region routing been strict), the second request would have been a HIT and rejected with the cached response.

Dependency health

Payment-service (us-east-1): healthy
Payment-service (us-west-2): healthy
Idempotency cache (per-region Redis): each healthy locally
Cross-region replication: DEGRADED (47s lag; network partial packet loss)
LB geo-routing: DEGRADED (DNS caching caused split routing)
Stripe/Adyen: healthy (they processed both auth requests successfully)
Merchant integrations: many showing 500 → retry pattern

Your investigation

You look at the dashboard. Which single metric tells you the most useful thing right now?

Hint: The customer symptom is duplicate charges. What upstream metric explains it?

60 seconds to decide a mitigation. What do you do RIGHT NOW?

Hint: You have two levers: (a) stop new duplicates, or (b) fix the network. Which is faster?

Primary-only routing works — duplicate rate drops to 0.03% within 5 min (baseline). You still have ~700 duplicate charges from the incident hour. What's your postmortem root-cause and the top action items?

Hint: The proximate cause was network partition + DNS split. What's the systemic failure?

In the retro, someone says 'this was really Stripe/Adyen's fault — they should have deduped the second auth.' Would that be a fair characterization?

Hint: Whose contract was broken?

In the retro, someone asks 'should we have caught this in staging?' Would we have? Why or why not?

Hint: What did staging look like when we tested cross-region? What's unique about a partition scenario?

Knowledge graph

Learn these first

Double-entry bookkeeping semantics (auditor-standard)
IETF Idempotency-Key HTTP header spec
Saga pattern + compensating actions
Cross-region consistency and split-brain scenarios

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.