Skip to main content
Back to Dropbox
MASTERCLASS
Gold-standard deep dive

Dropbox — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001
Architecture Decision Record

Custom-built storage tier ('Magic Pocket') on commodity hardware with Reed-Solomon erasure coding + SMR disks + content-addressed block dedup (chosen over pure-S3, pure-block-storage, or hybrid)

Context
We are Dropbox operating at ~700M registered users, 100+ PB of stored data, 4.5B files uploaded daily. Storage cost is our dominant infra expense. From 2007-2016, we ran on AWS S3. In 2016, we completed a 2-year 'Infrastructure Optimization' project moving 90%+ of our data OFF S3 onto our custom-built storage tier (Magic Pocket) in our own datacenters. This decision was one of the largest cloud-repatriation projects publicly discussed and saved Dropbox ~$75M/yr documented. The tradeoff space: massive capex + hardware team vs. AWS S3's pay-per-GB model at scale.
Constraints
  • 700M+ registered users, ~15M paying customers
  • 100+ PB stored data; 500 PB+ raw storage (with replication + erasure coding)
  • 4.5B files uploaded/day; ~1B active file syncs/day
  • P99 upload latency: < 200 ms for small files; < 5 seconds for 100 MB files
  • Availability: 99.99% at file-read; durability: 11+ 9s (data-loss annual probability < 10⁻¹¹)
  • Cost target: $/GB stored must beat AWS S3 by 40%+ at our scale for the business to be viable at consumer pricing
  • Compliance: SOC 2, GDPR data-residency (EU users' data stays in EU)
  • Team: ~50 engineers on the storage team (out of ~2500 total)
Options considered

Custom Magic Pocket + Reed-Solomon + SMR disks + content-addressed dedup (chosen)

Pros
  • 40-70% cost savings vs. AWS S3 at our scale — documented $75M/yr savings
  • Full control over the storage stack — can optimize for our workload's write patterns
  • SMR disks (shingled magnetic recording) give ~25% higher storage density at lower cost — a technology only viable if you control your fleet
  • Reed-Solomon erasure coding (10+4 configuration) gives 11+ 9s durability at 40% overhead vs. 3× replication
  • Content-addressed dedup at block level saves 30-50% storage across the fleet (users store the same photo, PDF, etc.)
  • Independent of any cloud provider — insurance against provider issues + pricing changes
Cons
  • Capex: ~$100M+ upfront for datacenter buildout + storage tier + software
  • Hardware team required — SMR drives, fleet management, physical operations
  • Migration from S3 was a 2-year project — extremely risky and expensive during transition
  • Any hardware bug (like SMR firmware issues) is our problem, not AWS's
Cost: $100M+ upfront + ongoing hardware refresh + ~50-person storage team. Payback in ~2 years at our scale.

Continue on AWS S3 (pre-2015 architecture)

Pros
  • Zero capex; no hardware team
  • S3 is battle-tested + globally available
  • AWS's engineering team handles all storage-tier innovation
  • Multi-region durability built-in
Cons
  • AWS S3 at our scale = $200M+/yr documented (Dropbox's 2016 filing)
  • Every scale-up commits us to more spending — no operating leverage
  • S3's write-latency SLA doesn't fit our small-file-sync workload perfectly
  • Data-residency requires multiple regional buckets + complex routing
Cost: $200M+/yr in AWS bills. Grows with our data, not our revenue.

Hybrid: hot tier on S3 + cold tier on custom storage

Pros
  • Fresh data on managed S3 (best UX for uploads)
  • Cold data on cheaper custom storage (best economics for the 90% that's never re-accessed)
  • Simpler than full migration
Cons
  • Storage-tier boundary is complex — cross-tier movement is a large ongoing engineering burden
  • Metadata complexity (tracking which tier each file lives on)
  • Doesn't capture the full cost savings — hot tier still expensive
  • Two operational stacks instead of one
Cost: 50-60% of pure-S3 cost, but doubles operational complexity.

Multi-cloud (S3 + GCS + Azure Blob) for redundancy

Pros
  • Hedged against single-cloud outages
  • Bargaining leverage on prices
Cons
  • 3× the ops complexity for modest reliability gains
  • Cross-cloud replication has egress costs (S3 → GCS = $$$)
  • Doesn't reduce cost; it multiplies it
  • Netflix and Dropbox both publicly rejected this model
Cost: 150-200% of pure-S3 cost. Punitive.
Chosen solution

Custom Magic Pocket + Reed-Solomon + SMR + block dedup (option 1)

Why
At Dropbox's scale, cloud egress + storage fees dominate infra economics. The $75M/yr savings pay for the storage team + hardware refresh many times over. AWS S3 remains excellent for the vast majority of companies, but Dropbox's very specific workload (deduplicable small-to-medium files stored long-term, read-cold) is exactly the workload where custom storage wins. Reed-Solomon 10+4 gives us 11+ 9s durability at 40% overhead (vs. 200% for 3× replication) — the math is superior. SMR drives give us 25% density gains. Block-level dedup saves another 30-50% (many users store the same PDF, photo, etc.). Each optimization is small individually; together they cross the S3-crossover threshold. The 2-year migration was risky and expensive — but the ongoing savings compound forever.
Rejected alternatives (with reasons)
  • Continue on S3 — $200M+/yr, growing with data not revenue
  • Hybrid — doubles ops complexity for partial savings
  • Multi-cloud — 150-200% cost for modest reliability gains
  • Continue with 3× replication instead of erasure coding — 200% overhead vs. 40% for RS
Trade-offs accepted
  • Accept the hardware + logistics + firmware complexity in exchange for cost + control
  • Accept a 2-year migration cost + risk to lock in ongoing savings
  • Accept SMR firmware issues (Dropbox has publicly discussed several) as part of controlling our fleet
  • Accept the AWS lock-in for other services (control plane still on AWS) — the storage tier is the piece where Dropbox-scale economics change
Consequences
  • 50-person storage team becomes a first-class engineering discipline
  • Datacenter operations become a core capability (physical, power, cooling, hardware refresh)
  • SMR firmware becomes an internal expertise — patches, bug reports, vendor negotiations
  • Block dedup pipeline becomes a first-class system with its own SLA (dedup rate, ratio, error handling)
  • Reed-Solomon repair operations run continuously — repair-shard reads must complete in bounded time
  • Cross-datacenter replication + regional failover become internal engineering (not AWS's job anymore)
When would we reverse this decision?
  • If AWS S3 pricing drops 60%+ (unlikely but possible), reconsider return to S3
  • If our workload shifts to heavily-accessed hot data (e.g., we pivot to a streaming product), Magic Pocket's cost advantage disappears
  • If regulatory landscape mandates specific-cloud-provider storage (some jurisdictions do), hybrid may return
  • If our engineering team shrinks such that the storage team is unsupportable, migration back to S3 is a multi-year project but possible

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the founding backend engineer at a startup building a specialized file-sync product for medical practices (secure PDF + image sharing between clinicians, HIPAA-compliant). The founders point at Dropbox and say 'we need that, but for healthcare.' You have 3 engineers, 12 weeks, and a hospital-partnership pilot on the calendar. Your target: 500 clinics × 20 clinicians/clinic × ~100 files/day = 1M files/day. Total data at year-end: ~30 TB. Founders love Dropbox's UX but haven't thought about HIPAA, dedup, or Magic Pocket.

Constraints
  • 112-week deadline (aligned with hospital pilot)
  • 23 backend engineers; no storage-team experience
  • 31M files/day × avg 5 MB = 5 TB/day of writes; ~30 TB stored at year-end
  • 4Existing stack: Rails + Postgres + Redis on AWS
  • 5Budget: dedicated storage infra < $8K/mo
  • 6HIPAA compliance: strict PHI handling, audit logs, BAA with any vendor
  • 7Dropbox-style features requested: sync across devices, shared folders, version history
  • 8Founders' pitch mentions 'Dropbox-quality sync with medical-grade compliance'
  • 9Regional: US-only initially, EU expansion year 2
Your question

What architecture do you propose, and how do you scope 'Dropbox-quality' into 12 weeks with a 3-engineer team? Be specific about storage backend, sync algorithm, HIPAA compliance, and the founder conversation.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT
Sync conflict storm — a user's 30 devices generate 12,000 conflicts in an hour

PagerDuty alert at 11:22 UTC. Support tickets are pouring in from one clinic (large hospital network). Their staff shared a single spreadsheet across 30 devices (30 doctors' iPads). Everyone's been editing it simultaneously. Our conflict-resolution system generated 12,000 conflict-copy files in the last hour for this ONE file. Support says 'the doctors are all seeing files-conflict-copy-Dr-Chen-2026-08-30-11-16, files-conflict-copy-Dr-Patel-2026-08-30-11-17, etc. cluttering their folder.' Trust score of the platform with this hospital is at risk. You are on-call.

Metrics
  • clinic.10247.conflict_files_created_last_hour: 12,000 (baseline: ~5/hour)
  • file.spreadsheet_XYZ.edit_events_last_hour: 4,800
  • user.concurrent_editors_of_file: 30 (Dr Chen, Dr Patel, Dr Sanchez, ...)
  • sync.conflict_resolution_algorithm: last-writer-wins + preserve-conflict-copy
  • storage.growth.clinic_10247: +50GB in last hour (from spreadsheet conflicts alone)
  • support.tickets.clinic_10247: 40 tickets in 30 min
  • app.customer_success_slack: EXPLODING (customer success team fielding calls)
  • spreadsheet_XYZ.avg_edit_time_between_conflicts: 90 seconds (multi-device users saving continuously)
Logs
  • 11:00 UTC — normal sync operations
  • 11:05 UTC — spreadsheet_XYZ (shift-schedule.xlsx) first opened by 30 doctors simultaneously
  • 11:07 UTC — first conflict created: Dr Chen and Dr Patel both edit line 5
  • 11:10 UTC — second conflict: Dr Sanchez and Dr Wong both add rows
  • 11:15 UTC — conflicts accelerating: 8 per minute
  • 11:20 UTC — conflict-creation rate hits 100/min
  • 11:22 UTC — PagerDuty alert fires (conflict_files_last_hour > 1000)
Traces
  • The conflict-generation cycle:
  • Time T: Dr Chen edits spreadsheet on iPad → local save → begin sync
  • Time T+2s: Dr Patel edits spreadsheet on their iPad → local save → begin sync
  • Time T+3s: Dr Chen's sync completes → server updates canonical version to Chen's
  • Time T+5s: Dr Patel's sync arrives → server detects concurrent-edit (base version from ~5s ago)
  • Time T+5s: server creates spreadsheet-conflict-copy-Patel-<timestamp>.xlsx
  • Time T+5s: server sends both files back to all 30 devices
  • Time T+6s: all 30 devices download BOTH files; Dr Chen's original + Patel's conflict
  • Time T+8s: Dr Sanchez sees the two files, decides to fix; edits Dr Chen's version
  • Time T+10s: Dr Sanchez's sync arrives → concurrent-edit vs. now-current version → ANOTHER conflict copy
  • Time T+12s: Dr Wong is editing Dr Patel's conflict copy... conflict copies of conflict copies begin appearing
  • Time T+30min: 12,000 conflict copies of the same underlying file
Dependency health
  • Sync engine: healthy (working as designed; the ALGORITHM is causing the storm)
  • S3 storage: healthy (absorbing the writes)
  • Notification system: healthy
  • Client apps: reporting 'sync complete' but folder is filling up
  • Customer trust: DEGRADED
Your investigation
1

The sync engine is 'working as designed.' What is the design flaw the doctors are exposing?

Hint: Last-writer-wins is standard for concurrent edits. But what's the failure mode when N people edit simultaneously?
2

60 seconds to decide. What do you do RIGHT NOW to stop the conflict-storm?

Hint: You need to (a) stop new conflicts + (b) not lose data. What's the fastest action?
3

Locking works — no new conflicts in the next hour. What's your postmortem root-cause and top action items?

Hint: The proximate cause was our sync algorithm on this workload. What's the systemic gap?
4

In the retro, someone says 'we should build real-time collaborative editing like Google Docs.' Would that be the right response?

Hint: That's a huge undertaking. Is it the right one?
5

Customer success asks 'should we refund the hospital's month?' What's your recommendation and framing?

Hint: Financial vs. relational tradeoff during a customer-visible incident.
Knowledge graph

Learn these first

  • Content-addressed storage + block dedup
  • Reed-Solomon erasure coding (10+4 configuration)
  • SMR (Shingled Magnetic Recording) drive characteristics
  • rsync-style delta sync algorithms
  • Operational Transforms (OT) vs. file-based sync semantics

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.