Skip to main content
Back to YouTube
MASTERCLASS
Gold-standard deep dive

YouTube — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001
Architecture Decision Record

Vitess-sharded MySQL for metadata + object storage (GCS/S3) for source video + async DASH/HLS multi-variant transcoding + global CDN with adaptive bitrate (over single-shard SQL, monolithic MP4-progressive, or single-codec H.264-only)

Context
We are building a UGC video platform. At maturity we expect 500+ hours of video uploaded per minute (YouTube's public number), 2.5B monthly logged-in users, 1B+ hours of video watched daily. Metadata is huge (billions of videos with per-video comments, likes, view counts, playlists), and MySQL sharding is essential. Video files must transcode into multiple resolutions × codecs (720p/1080p/4K × H.264/H.265/VP9/AV1) to serve every device on every network condition. CDN egress dominates infra cost. Live streaming is a first-class product surface (World Cup, election night, product launches).
Constraints
  • 500+ hours uploaded/min at YouTube maturity; 2.5B MAU
  • 1B+ hours watched/day; peak concurrent viewers during global events ~100M
  • Availability: 99.99% for playback; 99.9% for upload; live streams have their own SLA (< 5s glass-to-glass latency for premium live)
  • Video quality: 240p → 4K variants for every video; multiple codecs for device compatibility
  • Storage cost: an average 10-min video with all variants totals ~2 GB stored → 500 h × 60 × 2 GB = 60 TB uploaded PER MINUTE
  • Metadata scale: billions of video rows; SELECT queries on 'videos I've watched' need sub-100ms
  • Live streaming: latency budget < 5s glass-to-glass at maturity (competing with Twitch + Netflix Live)
  • Copyright compliance: Content ID scan every upload against a fingerprint database of billions of tracks + videos
Options considered

Vitess-sharded MySQL + GCS/S3 source storage + async multi-variant transcoding + global CDN with adaptive bitrate (chosen)

Pros
  • Vitess (open-sourced by YouTube in 2012) is battle-tested MySQL sharding — solves the billions-of-videos metadata problem
  • Object storage (GCS at YouTube) gives durable + cheap source video storage
  • Async transcoding lets us handle 500 h/min upload burst without user-visible latency
  • Multi-codec + multi-resolution (adaptive bitrate) serves every device from 2G-phone to 4K-TV
  • Global CDN with adaptive bitrate is the industry pattern for scaled video (Netflix Open Connect, YouTube edge, Twitch)
  • DASH + HLS both supported: DASH is the modern standard; HLS is required for iOS/Safari
Cons
  • Vitess ops burden is significant — resharding, hot-shard detection, and query routing all require specialized SREs
  • Async transcoding means uploaders wait 5-30 min before their video is playable in all variants
  • Multi-codec transcoding is CPU-intensive; storage cost balloons (each video stored ~10× because of variants)
  • Live streaming requires a completely separate origin-egress stack — cannot share the async-transcode pipeline
Cost: ~$50M/yr just for transcoding + storage at YouTube-scale steady-state. CDN egress dwarfs this (10× or more).

Single-shard MySQL / monolithic DB (like early YouTube pre-Vitess)

Pros
  • Simpler operational model
  • Native SQL semantics
Cons
  • Cannot scale past a few million videos before write throughput becomes the bottleneck
  • This is exactly the reason YouTube built Vitess in the first place
  • Migration off single-shard MySQL takes 6+ months
Cost: Deceivingly cheap upfront; catastrophic at scale.

MP4-progressive delivery (single file per video, no adaptive bitrate)

Pros
  • Simplest delivery model
  • Works out of the box in every browser
Cons
  • No quality adaptation — 2G users get 4K video (unwatchable) or 4K users get 240p (bad UX)
  • No seeking optimization — jumping to minute 5 downloads bytes 0-5 first
  • CDN cost balloons because we serve the SAME variant to everyone regardless of connection
Cost: 5-10× the CDN egress cost of adaptive bitrate at the same user satisfaction.

H.264-only single codec (simplify the codec matrix)

Pros
  • Simplest transcoding pipeline
  • H.264 is universally supported
Cons
  • H.264 is ~40% larger than H.265 at same quality → 40% more CDN egress
  • AV1 is even better (~50% smaller than H.264) — leaving it on the table costs billions in egress fees
  • Some devices only support specific codecs (older TVs, embedded devices) — H.264 alone leaves them serving suboptimal quality
Cost: 40-50% higher CDN egress fees at video-heavy scale.
Chosen solution

Vitess + GCS + multi-variant + adaptive bitrate CDN (option 1)

Why
At YouTube's scale, every alternative is prohibitive. Single-shard SQL was YouTube's original architecture — they built Vitess specifically because they hit its limits. MP4-progressive doubles CDN egress fees (which are the dominant cost line). H.264-only leaves 40% egress savings on the table. The chosen shape is the industry-consensus answer for UGC video at scale, with a clear reason for every decision: sharding for metadata scale, object storage for source cost, multi-variant transcoding for device compatibility, adaptive bitrate for network condition variety, global CDN for latency + egress optimization. Every major video platform (YouTube, Twitch, TikTok, Facebook Video) converges on this pattern with vendor-specific variations.
Rejected alternatives (with reasons)
  • Single-shard MySQL — YouTube's own history documents why this doesn't work at scale
  • MP4-progressive — 5-10× CDN egress cost at same user satisfaction
  • H.264-only — leaves 40% egress savings on the table
  • Skip live streaming (do only VOD) — competitive dead-end; every serious video platform now has live
Trade-offs accepted
  • Accept Vitess operational complexity (resharding, hot-shard detection) in exchange for horizontal metadata scale
  • Accept 5-30 min upload-to-fully-playable-in-all-variants latency — the alternative (sync transcoding) kills the upload UX
  • Accept multi-codec + multi-resolution storage cost — the CDN egress savings pay for the storage 5-10×
  • Accept the separate live-streaming pipeline — VOD and live have fundamentally different SLAs
Consequences
  • Vitess SRE team becomes a first-class discipline (hire specialists)
  • Every video upload triggers 8-16 transcode jobs (variants × codecs × resolutions)
  • CDN routing becomes a business function — negotiating with 3+ CDN providers annually
  • Live streaming gets its own architecture, its own on-call, its own capacity planning
  • Content ID scan runs on every upload, blocking publication for flagged content
When would we reverse this decision?
  • If a new managed video service emerges with feature parity + significantly lower cost, revisit the buy-vs-build
  • If AV1 adoption reaches near-universal (currently ~40%), simplify codec matrix by dropping H.264
  • If our workload shifts to short-form only (TikTok-style), the adaptive-bitrate complexity is overkill
  • If regulatory landscape mandates specific content-scan mechanisms, Content ID architecture must adapt

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the founding backend engineer at an EdTech startup building a video-hosting product for teachers (think 'Vimeo for K-12 educators'). The founders want to 'ship a YouTube-quality video hosting product' for their MVP: teachers upload lesson videos, students watch, teachers can post live-streamed office hours. You have 3 engineers, 12 weeks, and a target of 10K teacher accounts × 200 students each = 2M users. Total video volume: ~1K uploads/day at MVP, growing to 50K/day in year 2.

Constraints
  • 112-week deadline (aligned with the September back-to-school season)
  • 23 backend engineers; none has built a video pipeline
  • 31K uploads/day at MVP; 50K/day year 2 (ceiling for planning)
  • 4Existing stack: Rails + Postgres + Redis + Sidekiq on AWS
  • 5Budget: dedicated video infra < $10K/mo
  • 6Live streaming (teacher office hours) is a P1 feature for founders — they want it in the MVP
  • 7Content is mostly 10-30 min lesson videos; live streams are 30-60 min
  • 8Compliance: FERPA (education privacy) is a hard requirement
  • 9Founders' pitch to VCs mentions 'YouTube-quality streaming with adaptive bitrate'
Your question

What architecture do you propose, and how do you scope the 'YouTube-quality' ambition into a 12-week ship? Be specific about VOD, live streaming, buy vs. build, and the founder conversation.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT
Live stream origin saturates during World Cup final — 40M concurrent viewers, CDN cache-miss storm

PagerDuty alert at 21:47 UTC during the World Cup final. Live stream player error rate spiking (users see 'video unavailable'). CDN cache-hit ratio has dropped from 98% to 62%. Origin server CPU pinned at 100%. Concurrent viewer count is 40M, ~5× higher than the previous non-final match. The event started 45 min ago. You are on-call.

Metrics
  • live.concurrent_viewers: 40M (was 8M for previous non-final matches)
  • cdn.cache_hit_ratio: 62% (was 98% baseline)
  • cdn.origin_pull_rate: 380K req/sec (was 5K req/sec — 76× baseline)
  • origin.cpu: 100% pinned across all origin nodes
  • origin.network_out: 4.2 Tbps sustained (was 300 Gbps)
  • player.error_rate: 8.4% (baseline 0.05%)
  • player.rebuffer_ratio: 12% (baseline 0.3%)
  • twitch.mentions_of_our_service: exponential; users on Twitter reporting 'stream keeps buffering'
  • abr.selected_variant.distribution: shifted heavily to 480p (players stepping down due to network stress)
Logs
  • 21:00 UTC — match starts; concurrent viewers climb from 500K → 8M within 5 min (expected)
  • 21:15 UTC — goal scored; viewer count spikes to 20M within 3 min
  • 21:20 UTC — CDN cache-hit begins dropping (86% → 74%)
  • 21:35 UTC — halftime; viewer count peaks at 32M
  • 21:40 UTC — halftime ends, second half starts; viewer count 40M (all-time record for our platform)
  • 21:45 UTC — CDN cache-hit drops below 70%
  • 21:47 UTC — PagerDuty alert fires (cache-hit < 90%)
Traces
  • Trace of a stream request during the storm:
  • → Player fetches HLS segment (e.g., 1080p_segment_5847.ts)
  • → Cloudflare edge cache MISS (segments have short TTL + high uniqueness)
  • → Origin pull request → our origin server
  • → Origin: read from Kafka live-stream buffer + repackage → serve
  • → 380K concurrent origin pulls → CPU pinned
  • → Latency 30-800ms depending on load
  • → Some players hit HTTP 502 (origin rejecting new connections)
  • The cache-miss cascade:
  • Live stream = every segment is new every ~4s → high cache-miss ratio by design
  • 40M viewers × 1 segment fetch every 4s = 10M segment reqs/sec at edge
  • Even at 98% cache-hit, that's 200K origin pulls/sec — barely tolerable
  • At 62% cache-hit, that's 3.8M origin pulls/sec — origin drowns
Dependency health
  • Player clients: healthy but reporting errors + rebuffers
  • CDN edge: healthy but cache-hit degraded
  • Origin server fleet: DEGRADED (CPU 100%)
  • Live stream ingest (RTMP): healthy
  • Live stream transcode fleet: healthy
  • Kafka live-stream buffer: healthy
  • Metadata / auth: healthy (viewers pre-authenticated)
  • Analytics + billing: healthy but delayed
Your investigation
1

You look at the metrics. What single insight explains the cache-miss cascade?

Hint: Live streaming segments are unique every ~4 seconds. What happens when 40M concurrent viewers all request slightly different segments?
2

You realize the underlying issue: variant fragmentation is spreading load across many segment files. What's your hypothesis for why cache-hit dropped from 98% to 62%?

Hint: The player adaptive-bitrate step-down during a stress event means viewers switch between quality levels. Each quality level is its own segment set.
3

60 seconds to decide a mitigation. What do you do RIGHT NOW?

Hint: You have two levers: (a) reduce origin load, or (b) reduce variant fragmentation. Which is faster?
4

Manifest change works — cache-hit climbs back to 89% within 3 min. What's the postmortem root-cause and top action items?

Hint: The proximate cause was ABR variant fragmentation. What's the systemic gap?
5

The team asks 'should we buy Mux/AWS Elemental Live instead?' What's the honest tradeoff at our scale?

Hint: Live streaming has different economics than VOD.
Knowledge graph

Learn these first

  • Adaptive Bitrate streaming (HLS + DASH manifest structure)
  • Vitess-sharded MySQL (YouTube's open-source contribution)
  • Live streaming vs. VOD architectural differences (cache locality)
  • Origin shield tier design (mid-tier caching for cache-miss cascades)

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.