Skip to main content
Back to Distributed Database
MASTERCLASS
Gold-standard deep dive

Distributed Database — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001
Architecture Decision Record

Leaderless multi-master with tunable consistency (Dynamo-style R+W>N + configurable replication factor) — over Spanner-style strong global consistency, single-leader with async replication, or CockroachDB-style Raft groups

Context
We are building a globally distributed database for a marketplace product. 500M+ users across 4 regions. Read:write is ~50:1. Peak: 5M reads/sec, 100K writes/sec. Consistency requirements vary by workload: user profile reads can be stale (100ms OK); order writes must be strongly consistent; analytics reads can be seconds-stale. We must survive: single-node failures (routine), datacenter-level failures (rare), regional failures (once/decade). Legal/compliance: some data must not leave certain jurisdictions.
Constraints
  • 500M+ users, 4 regions (NA-East, NA-West, EU, APAC)
  • 5M reads/sec + 100K writes/sec peak
  • Availability: 99.99% globally (< 52 min/year outage tolerated)
  • Durability: 11+ 9s (data loss annual probability < 10⁻¹¹)
  • Latency: p99 read < 50ms local; p99 write < 100ms local; cross-region reads accepted at 200-500ms
  • Consistency: per-table configurable (order data = strong; profile data = eventual)
  • Regulatory: GDPR (EU users' data stays in EU), various data-residency laws
  • Team: 20 engineers on database platform; hiring specialists is hard
Options considered

Leaderless multi-master Dynamo-style with tunable R+W (chosen)

Pros
  • Every node accepts writes — no coordinator bottleneck; near-linear write scale
  • R+W>N invariant gives correct behavior: R=2,W=2,N=3 = strong; R=1,W=1,N=3 = eventual
  • Regional failures don't cascade — each region continues serving from local nodes
  • Tunable per-table: order table = W=all,R=1 (write-heavy correctness); profile table = W=1,R=1 (fast eventual)
  • Battle-tested at Amazon (DynamoDB), Cassandra (Meta/Netflix/Uber), Riak (Yandex)
  • Vector clocks handle conflict resolution deterministically
Cons
  • Eventual consistency semantics are non-obvious to app developers — bugs happen
  • R+W>N invariant is a real correctness burden — misconfiguration = data loss
  • Anti-entropy repair (Merkle tree gossip) has ongoing overhead
  • Multi-master conflict resolution requires per-table strategy (last-write-wins, vector-clock, CRDT)
  • Cannot serve interactive transactions across many keys — no distributed transactions
Cost: 20 engineers ongoing; ~$500K/mo infra at 500M users; complexity is real ongoing cost.

Spanner-style with TrueTime + Paxos groups

Pros
  • External consistency = you can reason about it like a single machine
  • ACID transactions across globally-distributed data
  • TrueTime bounds clock uncertainty (typically <10ms)
  • Battle-tested at Google (Spanner backs Gmail, Ads, YouTube analytics)
Cons
  • Requires atomic-clock + GPS infrastructure OR must accept 100+ ms latency to bound uncertainty
  • Complex operationally — Paxos groups, split/merge, hot-shard rebalance
  • Write latency: 10-100ms typical (2-phase commit + Paxos rounds)
  • Cannot survive region-level failures without split-brain if TrueTime is lost
Cost: Google-scale operational discipline; hard to hire ex-Google engineers.

Single-leader (MySQL/Postgres with read replicas + async replication)

Pros
  • Familiar semantics (single master; readers see consistent snapshots)
  • Easy to reason about
  • Team already knows how to operate
Cons
  • Single-leader write bottleneck — 100K writes/sec is beyond a single primary
  • Regional failover requires manual coordination + accepts data loss window
  • Cross-region reads pay full latency; no way to serve from local replica
  • Cannot scale reads AND writes independently
Cost: Cheap upfront; hits a hard wall at 100K writes/sec.

Buy DynamoDB Global Tables (managed)

Pros
  • Fully managed — no operational burden
  • Multi-region replication built in
  • Battle-tested at Amazon retail + AWS's own services
Cons
  • Vendor lock-in on the most critical system
  • Cost: $12-50K/mo at 5M reads/sec + 100K writes/sec = ~$3-15M/yr at our scale
  • Cannot customize consistency model (Dynamo tunable is available but different from OUR tunables)
  • Data-residency requires per-region tables — cross-region access adds complexity
Cost: $3-15M/yr at our scale.
Chosen solution

Leaderless multi-master with tunable R+W (option 1)

Why
At our scale (5M reads/sec, 100K writes/sec, 4 regions), we need: (a) writes to succeed even if a whole region is down, (b) per-table configurable consistency (order = strong, profile = eventual), (c) linear scale on both reads + writes, (d) < 50ms local latency. The chosen shape (Dynamo-style leaderless) delivers all four. Spanner is over-engineered for our use case (we don\'t need external consistency for user profiles). Single-leader can't handle 100K writes/sec globally. Buying DynamoDB is $3-15M/yr + vendor lock-in. Our own leaderless architecture on commodity hardware is 2-4× cheaper + gives us configurability. The complexity is real but manageable — Cassandra/Riak have proven this at Netflix/Uber/Meta scale.
Rejected alternatives (with reasons)
  • Spanner-style — over-engineered for our workload; requires atomic-clock infra we don't have
  • Single-leader — write bottleneck at 100K/sec
  • DynamoDB Global Tables — $3-15M/yr + lock-in
  • SQL-only (Postgres with sharding) — write bottleneck + no native multi-region
Trade-offs accepted
  • Accept eventual consistency for most workloads — educate app developers on semantics
  • Accept anti-entropy repair overhead — background gossip + Merkle tree comparison
  • Accept per-table conflict resolution strategy — LWW default, CRDT for counters, vector clock for complex cases
  • Accept 'no distributed transactions across many keys' — design apps around single-key + eventual reconciliation
Consequences
  • Database SRE team becomes a first-class discipline (specialists needed)
  • App developers must learn tunable consistency (per-table + per-query configuration)
  • Reconciliation jobs run continuously — anti-entropy + read repair
  • Every table declares its own consistency + conflict-resolution strategy
  • Hot-shard detection + auto-rebalancing become operational disciplines
When would we reverse this decision?
  • If we adopt Spanner-quality atomic-clock infra, revisit strong-consistency option
  • If our workload consolidates to a few very-critical tables, single-leader with strong consistency may suffice
  • If DynamoDB pricing drops significantly, buy becomes attractive again
  • If a new class of distributed DB emerges (CockroachDB, TiDB, YugaByte reached scale-parity), evaluate

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the CTO at a Series B SaaS startup. Your product is a real-time collaboration tool. Traffic: 10M documents, 1M active users, ~50K queries/sec. Existing stack: Postgres Multi-AZ. Recent performance issues have you thinking about the next step. Your board is pushing 'we should upgrade to a distributed database like Spanner or CockroachDB before we hit scale.' You have 4 engineers, 12 weeks between product roadmap items to consider this.

Constraints
  • 112-week evaluation window
  • 24 backend engineers; none with distributed-database operational experience
  • 350K queries/sec current; growing 20% quarterly
  • 4Data size: 500 GB currently; growing 40% annually
  • 5Existing Postgres: db.r6g.4xlarge Multi-AZ; ~$3K/mo
  • 6Budget: <$15K/mo for the database tier
  • 7Board's request: 'evaluate Spanner or CockroachDB'
  • 8Current Postgres CPU utilization at peak: 65% (not saturated yet)
  • 9Team is comfortable with Postgres; no NoSQL / distributed-DB experience
Your question

What do you recommend to the board, and how do you evaluate 'Spanner or CockroachDB' honestly? Be specific about scaling path, costs, and the conversation.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT
Split-brain during cross-region network partition — R+W>N violated for 8 minutes

PagerDuty alert at 03:14 UTC. Cross-region network between US-East and EU is experiencing 60% packet loss (fiber cut in transatlantic cable, ~5 min ago). Our leaderless database (RF=3 across regions) is behaving unpredictably: some writes are being 'quorum-accepted' by only US-East nodes (which think 2 of 3 nodes have persisted); some reads from EU are returning older values. Data-integrity alarm firing. You are on-call.

Metrics
  • network.us-east.eu.packet_loss: 60% (baseline < 0.01%)
  • db.replication_lag.us-east_to_eu: 40s (baseline < 1s)
  • db.replication_lag.eu_to_us-east: 42s
  • db.write_quorum_success_rate: 92% (baseline 99.9%) — some writes are TIMING OUT waiting for cross-region ack
  • db.reads.stale_value_reported: 0.4% (baseline 0.001%) — customers seeing old data
  • db.consistency_config.orders_table: W=2, R=2, N=3 (should give strong consistency)
  • db.consistency_config.profile_table: W=1, R=1, N=3 (deliberately eventual)
  • app.support_tickets_last_hour: 340 (baseline 20)
  • customer_complaints: 'I updated my order status, then reloaded, and it went back to old status'
Logs
  • 03:00 UTC — normal operations
  • 03:09 UTC — network monitoring reports packet loss on transatlantic link
  • 03:12 UTC — first customer report: 'my order was updated, then it reverted'
  • 03:12 UTC — DB write timeouts begin appearing (waiting for cross-region ISR)
  • 03:13 UTC — data-integrity alarm fires
  • 03:14 UTC — PagerDuty pages you
Traces
  • Trace of a split-brain write scenario:
  • Customer in NA-East updates their order status at 03:11:07
  • Order write with W=2 (needs 2 nodes to ack)
  • Coordinator writes to Node A (US-East) → ACK
  • Coordinator writes to Node B (US-East backup) → ACK
  • Coordinator writes to Node C (EU) → TIMEOUT (packet loss)
  • → But W=2 requirement was met by 2 US-East nodes → returned SUCCESS to app
  • → App told customer 'order updated'
  • Meanwhile, customer in EU reads their order at 03:11:15
  • Read with R=2
  • Reads from Node C (EU) — has OLD value (write didn't propagate) → returns old
  • Reads from Node D (EU backup) — also has old value → returns old
  • → Two 'consistent' EU reads returning OLD value; R=2 satisfied
  • → Customer sees 'order status: PROCESSING' (should be COMPLETED)
  • The R+W>N invariant is 2+2>3 = 4>3 ✓ mathematically. But during a network partition, the 'quorum' can be satisfied within each partition SEPARATELY — leading to split-brain semantics.
Dependency health
  • US-East database nodes: healthy (accepting writes)
  • EU database nodes: healthy (serving reads, but stale)
  • Transatlantic network: DEGRADED (60% packet loss)
  • Application servers: healthy
  • Reads returning stale values: yes (0.4% of reads)
  • Writes timing out: yes (~8% of writes)
  • Customer trust: DEGRADED
Your investigation
1

What is happening architecturally?

Hint: The R+W>N invariant is mathematically true. Why is split-brain happening?
2

60 seconds to decide a mitigation. What do you do?

Hint: You can (a) sacrifice availability by rejecting writes until network heals, or (b) accept staleness as expected. Which?
3

Read-only mode works — data integrity restored in EU (though writes rejected). Network heals 3 hours later. What's the postmortem and top action items?

Hint: The proximate cause was network partition. What are the systemic gaps?
4

In the retro, someone says 'we should use Spanner — it gives strong consistency even under partition.' Would that solve this incident?

Hint: What does Spanner actually do during network partitions?
5

How do you communicate this externally?

Hint: The customers who saw stale data are angry. What's the honest comm?
Knowledge graph

Learn these first

  • CAP theorem + PACELC + eventual consistency
  • R+W>N quorum semantics + configurable consistency
  • Vector clocks + CRDTs for conflict resolution
  • Sharding strategies + hot-shard mitigation

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.