MASTERCLASS

Gold-standard deep dive

Netflix — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001

Architecture Decision Record

Custom Open Connect appliances deployed inside ISP networks + AWS for control plane + Chaos Engineering as ongoing discipline (over pure-CDN or pure-cloud-region-based delivery)

Context

We are Netflix at 260M+ subscribers, ~15% of global downstream internet traffic during peak hours. Our workload is video streaming (VOD-heavy, live increasingly relevant). We deliver ~150M hours of content daily. Our content library is CURATED (licensed premium + originals) — we know exactly which titles will be popular before we deliver them. This predictability changes the CDN economics. Our infrastructure decision from ~2011 was: (1) build our own edge appliances (Open Connect) and place them inside ISP networks, (2) run our control plane on AWS after migrating from our own datacenter (2016), (3) practice chaos engineering as ongoing discipline (Chaos Monkey, Chaos Kong, Simian Army).

Constraints

260M+ subscribers globally, ~15% of internet downstream traffic during peak hours
~150M hours of content watched daily; ~7.5B hours/quarter
4K + HDR + Atmos audio for premium content; 240p → 4K variants for every title
Availability: 99.99% at the subscriber-visible layer; regional AWS failures happen and must be routine
Content library: ~15,000 titles at any time; ~2 TB of encoded video per title (all variants) → ~30 PB total library
Predictable popularity: within days of a title's release, we know its viewing curve; can pre-position
Peak concurrent viewers: ~50M during premiere events; individual episodes can hit 30M+ concurrent
ISP relationships: Netflix has bilateral peering + Open Connect placement agreements with hundreds of ISPs globally

Options considered

Custom Open Connect appliances + AWS control plane + chaos engineering (chosen)

Pros

Open Connect delivers 95%+ of Netflix traffic FROM WITHIN the ISP's network — the ISP loves it (they save transit costs), and Netflix loves it (near-zero egress fees + lowest possible latency)
AWS control plane gives infinite compute + fast iteration for microservices without hardware provisioning
Chaos engineering (Chaos Monkey → Chaos Kong → FIT) tests failure modes in production continuously — outages become routine + non-events
Pre-positioning: we push tonight's popular content to Open Connect boxes overnight during off-peak, cache hits climb to 99%+ at peak
Independent of any single cloud vendor — control plane on AWS, edge on our own hardware, insurance against provider issues

Cons

Building + shipping + maintaining custom hardware appliances requires a hardware team + logistics team + firmware engineers
ISP relationships require dedicated business-development staff to negotiate placement agreements
Chaos engineering discipline requires cultural change — engineers must embrace 'we deliberately break things in production'
AWS lock-in for control plane creates counterparty risk (2013 Christmas Eve outage nearly killed Netflix streaming for the peak day)

Cost: Open Connect capital cost is significant (~$100M+ over the years) but pays for itself via ~$1B+/yr saved CDN egress fees.

Pure third-party CDN delivery (Akamai / Cloudflare / Level3)

Pros

Zero hardware team needed
Rapid geographic expansion — the CDN is already global
No ISP negotiation needed

Cons

CDN egress fees at Netflix scale would be ~$1-2B/yr direct cost — an existential burden
CDN providers have their own SLAs and priorities — a Cloudflare outage affects millions of Netflix subscribers
Pre-positioning requires CDN-specific APIs; Netflix's title-level popularity predictions can't be fully exploited
Netflix has been public that this was the initial architecture and they moved off it as scale forced them to build Open Connect

Cost: $1-2B/yr in CDN egress fees at Netflix scale. Punitive.

Pure cloud-region-based delivery (AWS/Azure/GCP regions as origins, ISPs pull from regions)

Pros

Simplest architecture — no custom hardware
Cloud provider handles everything

Cons

AWS/Azure/GCP egress fees are ~$0.05-0.09 per GB — at Netflix scale = billions/yr
Regional origin capacity would need massive expansion to serve 15% of global internet traffic
Cloud providers have latency floors (100-300ms cross-region) that competitive premium quality can't tolerate

Cost: Would require 3-5× current cloud spend without solving egress or latency.

Peer-to-peer / hybrid CDN (BitTorrent-flavored, peers help serve to each other)

Pros

Distributed load; no single point of failure
Free capacity from viewers themselves

Cons

Legal risk — copyright holders view P2P skeptically; licensing terms often forbid
Quality inconsistent — peers with weak upload bandwidth degrade experience for others
User acceptance for a paid subscription: users don't want their bandwidth used by Netflix

Cost: Low compute cost but legal + UX + brand cost is high.

Chosen solution

Open Connect + AWS control plane + chaos engineering (option 1)

Why

At Netflix's scale, egress economics dominate. Third-party CDNs are prohibitively expensive. Cloud-only is even more expensive. Peer-to-peer has licensing + UX problems. Open Connect is the only architecture that turns the ISP relationship into a mutual win: ISPs save transit fees (which they love) AND get better subscriber experience (Netflix content is fast) AND Netflix gets 95%+ traffic served with near-zero egress. AWS for the control plane gives us infinite compute for microservices + fast iteration. Chaos engineering is the discipline that lets us tolerate the inevitable AWS + Open Connect + regional failures without user-visible impact. This is the pattern that has scaled from ~30M subscribers (2013) to 260M+ (today).

Rejected alternatives (with reasons)

Pure third-party CDN — $1-2B/yr in egress; existential
Pure cloud origin — 3-5× current spend without solving latency
P2P hybrid — licensing risk + UX regression
Ship Netflix as a set of static video files (no streaming) — non-starter as premium video service

Trade-offs accepted

Accept hardware + logistics team overhead — pays for itself many-times-over
Accept ISP business-development complexity — the ISP relationships are moats
Accept AWS lock-in for control plane — insurance would require multi-cloud, and Netflix has publicly evaluated + rejected it
Accept the cultural change to embrace chaos engineering — this is what makes 'AWS us-east-1 is down' a non-event for us

Consequences

Hardware engineering + firmware become first-class disciplines at Netflix
ISP business-development becomes a permanent business function
Chaos Monkey (2011) → Chaos Kong (regional failure) → FIT (Failure Injection Testing) become the norm
Every architectural decision must consider 'what breaks when AWS us-east-1 goes down?' — the discipline that avoided catastrophe in 2016 + 2021
Open Connect placement negotiations become a business-development function (dedicated deal team)

When would we reverse this decision?

If CDN economics shift dramatically (e.g., a new CDN provider offers Netflix-scale pricing), reconsider Open Connect capex
If ISPs consolidate + demand higher placement fees, revisit the ISP model
If a new streaming protocol (like WebRTC-based ultra-low-latency) becomes competitive, our architecture must adapt
If we lose the premium subscription business (become an ad-supported product), the economic model shifts — Open Connect becomes overkill

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the tech lead at a Series C startup building a subscription video-on-demand service for a specific vertical (say, premium documentaries for a niche audience — 500K subscribers globally). Your founders point at Netflix and say 'we want that.' They've read a Netflix engineering blog post about Open Connect and mention 'we should build our own CDN eventually.' You have 4 backend engineers, 12 weeks, and no hardware experience. Your projected traffic in year 2: 500K subscribers × 3 hours/week × 5 GB = ~30 TB/day of egress.

Constraints

112-week deadline (aligned with launch announcement)
24 backend engineers; none has done hardware / CDN infrastructure
3500K subscribers projected year 2; 30 TB/day of egress
4Existing stack: Rails + Postgres + Redis + Sidekiq on AWS + Vercel frontend
5Budget: dedicated video infra < $15K/mo
6Content: ~800 documentary titles at maturity, each ~2 GB of variants encoded
7Founders' pitch to VCs mentions 'Netflix-quality streaming with our own CDN eventually'
8Premium subscribers expect 1080p at minimum, 4K for tent-pole titles
9Global audience — subscribers in NA, EU, APAC

Your question

What architecture do you propose, and how do you honestly translate 'Netflix-quality with our own CDN' into a 12-week ship? Be specific about CDN choice, encoding pipeline, cost, and the founder conversation.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT

AWS regional outage on Christmas Eve — control plane goes down, streaming must continue

PagerDuty alert at 15:30 UTC on December 24. AWS US-East-1 is experiencing a widespread outage — EC2 API failing, ELB routing degraded, RDS Multi-AZ failover failing. Multiple AWS services are impacted. Our control plane runs in US-East-1. We are seeing: user sign-in failures (10% of new sign-ins fail); subscription changes failing; recommendation service degraded. Streaming itself (video playback from Open Connect boxes to already-signed-in users) is HOLDING — because that path doesn't go through AWS at all. It's Christmas Eve. Our concurrent viewer count is at 40M and climbing. You are on-call.

Metrics

signup.success_rate: 90% (was 99.7%)
signin.new_session.failure_rate: 12% (was 0.3%)
subscription.change_api.failure_rate: 68% (writes failing to RDS)
recommendations.p99_latency: 8s (baseline 200ms)
streaming.session_stability: 99.8% (baseline 99.9%) — actually holding pretty well
openconnect.request_rate: 1.2M req/sec (peak season baseline)
aws.us_east_1.incident_severity: CRITICAL (AWS acknowledged widespread outage 15 min ago)
aws.eu_west_1_and_us_west_2: healthy — could serve requests but not currently the primary
chaos.kong.test_history: last full-region failover drill: 2 weeks ago; passed

Logs

15:15 UTC — normal operations
15:20 UTC — AWS US-East-1 acknowledges EC2 API issues
15:25 UTC — our SREs notice RDS failovers not completing
15:28 UTC — signup service starts returning 500s (RDS writes failing)
15:30 UTC — PagerDuty alert fires (signup success rate < 95%)
15:32 UTC — you get paged
15:35 UTC — AWS status page acknowledges widespread US-East-1 impact
15:38 UTC — internal Slack: 'is this the moment for Chaos Kong regional failover?'

Traces

Trace of a signup attempt at 15:28 UTC:
→ User submits form on client
→ CloudFront → API Gateway (US-East-1) — HEALTHY
→ API Gateway → signup Lambda — HEALTHY
→ signup Lambda → RDS write to Aurora us-east-1 — TIMEOUT (2000ms)
→ Lambda returns 500 to user
Trace of a video-play by an already-signed-in user (which is holding):
→ User's client already has an auth token from before the outage
→ Client requests video manifest from Netflix API (US-East-1) — RETRY FAIL, then fallback to cached manifest
→ Client requests video segments from Open Connect box inside their ISP — HEALTHY (Open Connect doesn't depend on AWS)
→ Playback continues normally
The critical insight: STREAMING (the actual watching) doesn't depend on our AWS control plane at all. But EVERYTHING new (signups, sign-ins, subscription changes, recommendations) does.

Dependency health

AWS US-East-1: CRITICAL (multiple services degraded)
AWS EU-West-1: healthy
AWS US-West-2: healthy
Open Connect global fleet: healthy (independent of AWS)
Cassandra (multi-region): partially degraded (US-East cluster affected)
EVCache: healthy (multi-region)
DynamoDB Global Tables (auth tokens): healthy
Client SDK offline behavior: authenticated users continue playing

Your investigation

You look at the metrics. What single insight explains everything?

Hint: Streaming is holding but everything new is failing. What does that pattern tell you?

The team debate: should we initiate Chaos Kong regional failover to US-West-2? What's the decision framework?

Hint: The tradeoff: fail over now (aggressive) vs. wait for AWS recovery (conservative).

Failover to US-West-2 works — signup success climbs back to 99% within 22 min. What is the postmortem and top action items?

Hint: The failover succeeded. But what would have made it faster or unnecessary?

In the retro, someone says 'this shows we should go multi-cloud.' Is that the right lesson?

Hint: Multi-cloud is a common instinct after cloud outages. Is it the right answer here?

How would you communicate this incident externally? Netflix subscribers are on Twitter reporting that the app 'is broken.'

Hint: Balance transparency + user reassurance.

Knowledge graph

Learn these first

Adaptive Bitrate streaming (HLS + DASH)
Open Connect economics (in-network delivery for major streaming services)
Chaos engineering discipline (Chaos Monkey, Chaos Kong, FIT)
Multi-region active-active failover patterns

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.