Dropbox: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Small file download — full cache hit (~70% of downloads)
User downloads a popular file (already cached at edge). Sub-100ms first byte, bandwidth-limited full download.
Auth + file ID.
File is chunked into 4MB blocks; metadata contains ordered list of block hashes.
Response with pre-signed block URLs.
4MB blocks fetched in parallel from CDN edge. Cache hit rate 95%+ for popular content.
Optimize: Parallel block fetching = full download completes at slowest block's speed. Bandwidth-limited.
Concatenate blocks in order. Local disk write.
Small file download is **~500-1500ms** — bandwidth-limited more than latency-limited. **Block-level parallelism** means 100MB file downloads at the aggregate speed of all block fetches. Sub-100ms metadata + parallel block fetch = smooth UX.
Bottleneck summary
Dropbox latency has THREE distinct profiles: **file download (bandwidth-limited, ~500-1500ms)**, **delta sync (only changed blocks travel, 100x bandwidth savings)**, **CRDT collaboration (~100-200ms per edit, own edits sub-frame)**. **The core architectural pattern**: block-level dedup + delta sync + own storage (Magic Pocket) transforms Dropbox from 'S3 wrapper' into 'genuinely differentiated storage platform'.
Optimization tips (this architecture)
- **Block-level dedup (4MB chunks)**: Chunk + SHA-256 hash. Blocks with same hash = same content = dedup. Massive storage savings.
- **Delta sync**: Only upload NEW blocks. 100x bandwidth savings for typical edits.
- **Magic Pocket + Reed-Solomon 14+4**: 22% storage overhead vs 200% triple-replication. $70M+/yr savings vs S3.
- **CRDT for collaboration**: Automerge/Yjs. Guaranteed convergent. No central coordination.
- **Parallel block download**: Fetch all blocks in parallel. Full download at aggregate speed.
- **Bloom filter on hash table**: Skip DB queries for definitely-missing hashes.
- **Local CRDT apply**: User sees own edit sub-frame. Remote apply eventually consistent.
- **Pre-signed URLs**: Direct-to-storage for large uploads. Bypass API bandwidth.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.