MASTERCLASS

Gold-standard deep dive

Instagram — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001

Architecture Decision Record

Async transcoding pipeline via Kafka + specialized transcode workers + multi-CDN egress (vs. synchronous upload-transcode or client-side transcoding only)

Context

We are building a photo + short-video sharing product (Instagram/Reels-style). Users upload media (photos 1-10 MB, videos 10-100 MB) and expect it to be viewable within seconds. At maturity, our workload is 100M+ MAU with ~50M photo uploads/day and ~5M video uploads/day. We must serve viewing at scale (multi-CDN global egress), maintain multiple transcoded variants per video (multiple resolutions × multiple codecs), and mitigate the viral upload spike — a trending moment can 10× normal upload volume within 60 seconds. We are on AWS with S3 for durable storage.

Constraints

Peak upload rate: 50M photos + 5M videos/day; viral peak 10× that within 60 seconds
Media viewing: 5B+ views/day across all posted media
Availability: 99.99% for viewing (CDN-tier); 99.9% for uploads (users accept brief queue at ingest under load)
Latency: photo view p99 < 200 ms globally; video first-frame p99 < 800 ms globally
Cost: multi-CDN egress at scale is the dominant infra bill — get this design wrong and gross margin collapses
Video transcoding is CPU-intensive: 30-90 seconds per minute of source video for full-quality H.265
Compliance: some jurisdictions require content-fingerprinting for copyright + CSAM at ingest before publication

Options considered

Async transcoding via Kafka + specialized workers + multi-CDN egress (chosen)

Pros

User's upload succeeds fast (client → S3 upload complete → 'processing' state)
Transcoding workers scale independently — burst capacity absorbs viral events without user-visible latency
Multi-CDN egress (Cloudflare + Akamai + Fastly) gives us bargaining leverage on egress fees + geo diversity
Kafka event bus lets downstream consumers (ML feature extraction, compliance scanning, analytics) subscribe without coupling

Cons

Multi-CDN is complex operationally — cache invalidation coordination across 3 CDNs is a small but permanent burden
Async 'processing' UX means users see a loading state for 10-90s on video uploads
Feature-extraction latency (for ranking features) adds a ~2-minute delay from upload to feed-eligible

Cost: ~$4M/yr at 50M photos + 5M videos/day (transcode fleet, multi-CDN egress, S3 storage).

Synchronous upload-transcode: user waits until transcoding completes

Pros

Simple UX — 'upload done' means transcoded and viewable
No async state machine to manage

Cons

Video uploads take 30-90 seconds — users see the loading spinner for a minute, huge UX regression
Peak concurrency at viral events becomes upload-tier concurrency (100K concurrent uploads waiting), overwhelming compute
Client-visible latency correlates directly with transcode fleet capacity — spiking fleet during viral events is expensive

Cost: Sync compute cost is ~50% higher than async because we cannot smooth the burst.

Client-side transcoding only (upload pre-transcoded from user's device)

Pros

Zero server-side transcoding cost
User's device does the work

Cons

Impossible to reliably enforce multi-resolution variants across the diversity of user devices
Battery drain on mobile — video transcoding is CPU-intensive and users notice
Cannot enforce copyright fingerprinting or CSAM scanning without server-side re-processing
Video codecs on user devices vary — some upload H.264, some HEVC, some AV1; server-side must homogenize

Cost: Server compute savings offset by mandatory server-side re-processing for compliance + variant generation.

Buy Mux/Cloudinary/Bitmovin (managed video pipeline)

Pros

Zero engineering effort — API-based video pipeline
Managed transcoding + CDN + analytics

Cons

Cost: $0.05-0.10 per minute of video transcoded × 5M videos/day × 30 sec avg = $150K-300K/day = $55-110M/yr at scale
Vendor lock-in on the media pipeline — the whole product depends on their SLA
Feature-extraction (for our ranking model) requires custom access that most managed services don't expose

Cost: $50-100M+/yr at scale. Punitive above 1M videos/day.

Chosen solution

Async transcoding pipeline + specialized workers + multi-CDN (option 1)

Why

At 5M+ videos/day, buying a managed pipeline becomes the single largest infra line item — 10× the cost of running our own. Client-side transcoding cannot solve the compliance + variant-generation problem. Synchronous upload-transcode kills the UX during viral spikes. The chosen shape gives us: (a) fast user-visible upload completion, (b) burst capacity via Kafka + worker fleet scaling, (c) multi-CDN bargaining power on egress fees (which dominate our bill), (d) an event bus for feature extraction + ML + compliance. Meta reportedly runs a similar pipeline at Instagram scale; publicly discussed at their engineering blog.

Rejected alternatives (with reasons)

Synchronous upload-transcode — kills UX during viral events + requires 50% more compute capacity to handle burst
Client-side-only — compliance + variant-generation invariants force server-side re-processing anyway
Buy Mux/Cloudinary — cost becomes prohibitive at video scale; also blocks our custom ML feature extraction
Single-CDN egress — no bargaining leverage; single-CDN outages become total-viewing outages

Trade-offs accepted

Accept the operational complexity of Kafka + worker fleet + multi-CDN coordination
Accept ~30-90s 'processing' UX for video uploads — the alternative is 30-90s spinner UX which is worse
Accept multi-CDN cache-invalidation complexity — the egress-cost savings pay for the ops burden 100×
Accept a ~2-minute feed-eligibility delay from upload — the alternative (feature-extraction on upload path) blocks the upload

Consequences

Kafka becomes a critical dependency for the whole media pipeline — SRE ownership becomes first-class
Every upload path must publish an event; downstream consumers subscribe without touching the upload service
Multi-CDN routing becomes a business function — bargaining renegotiates every 12 months as egress prices shift
Video transcode fleet scales dynamically to viral spikes — autoscaling based on Kafka lag becomes a discipline
'Processing' status becomes a first-class UX state — clients poll, backend publishes progress events

When would we reverse this decision?

If our video volume drops below ~500K/day sustainably, buying Mux becomes cheaper — revisit
If a new managed service emerges with feature-extraction API access + lower cost, revisit the buy-vs-build
If we launch a real-time video product (live streaming), the async pipeline needs to be paralleled by a low-latency streaming pipeline
If egress-fee negotiation stops mattering (some future CDN commoditization), collapse to single-CDN and save ops burden

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the founding backend engineer at a startup building a photo + short-video app for a specific niche (say, fitness content — 'Strava meets Instagram for gym-goers'). The founders point at Instagram and say 'we want that.' They demo'd a fitness-branded UI with algorithmic feed + Reels-style short videos + Stories. They have not thought about media pipeline cost, feed ranking cost, or multi-CDN egress. You have 2 engineers, 12 weeks, and 10K users with mean 15 followees.

Constraints

112-week deadline (aligned with investor demo)
22 backend engineers; neither has built a media pipeline at scale
310K existing users with ~15 mean followees; expected to grow to 100K by end of year
4Existing stack: Rails + Postgres + Redis + Sidekiq on AWS; using AWS S3 for user avatars only so far
5Budget: dedicated media + feed infra < $6K/mo (product is bootstrapped)
6Founders' pitch to investors mentions 'Instagram-quality feed with ML ranking'
7Compliance: no CSAM concerns at this vertical (gym-going content); still need copyright fingerprinting later
8Founders have not thought about video transcoding at all — every upload will be assumed to 'just work'

Your question

What architecture do you propose, and how do you honestly translate 'Instagram-quality feed' into a 12-week ship? Be specific about media pipeline, feed ranking (or absence thereof), cost, and the founder conversation.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT

Viral moment: video transcode pipeline saturates during a trending sports moment

PagerDuty alert at 22:37 UTC (major sports championship final just ended). Kafka video-transcode topic consumer lag has spiked from 500ms to 18 minutes. Transcode worker fleet at 100% CPU across all instances. Users report 'my video has been uploading for 45 minutes.' 12,000 videos are queued for transcoding. Upload rate is at 8× baseline. You are on-call.

Metrics

kafka.video_transcode_topic.consumer_lag: 1,080 s (baseline 500ms)
kafka.video_transcode_topic.production_rate: 40K videos/min (baseline 5K/min) — 8× baseline
kafka.video_transcode_topic.queue_depth: 12,000 videos pending
transcode_worker.pool.cpu_avg: 100% (fleet at 400 workers, all saturated)
transcode_worker.pool.count: 400 (as configured for baseline)
s3.uploads.rate: 8× baseline (users still uploading successfully — S3 handles it)
cdn.egress.rate: 3× baseline (existing videos being re-shared virally)
app.video_status.processing_count: 12,000 (baseline ~200)
app.upload_completion_status: 'processing' displayed for videos > 5 minutes old — user complaints climbing
sports.event_context: championship-final ended 2 min before spike began

Logs

22:34 UTC — normal upload rate
22:35 UTC — championship final ends; social sharing surge begins
22:35 UTC — video upload rate climbs from 5K/min to 10K/min
22:37 UTC — upload rate at 40K/min (8× baseline); Kafka production rate matches
22:37 UTC — transcode workers all at 100% CPU (~100 concurrent transcodes per worker)
22:37 UTC — consumer lag alarm fires (>60s)
22:38 UTC — you get paged

Traces

Trace of a video upload during the storm:
→ user's device uploads video to S3 via presigned URL (fast, ~2 sec for 20 MB file)
→ S3 upload complete webhook → app enqueues to kafka video-transcode-topic
→ app tells user: 'processing' state
→ 22:37 spike: 12,000 videos queued ahead of this one
→ transcode worker takes ~45 sec per video at fleet-of-400 concurrency
→ 12,000 / 400 = 30 waves of 400 videos each = 30 × 45s = 22 minutes for the current queue tail to complete
→ users see 'processing' for 22+ minutes; feed doesn't show their video
Contrast (baseline): queue depth = 200; 200 / 400 = 0.5 waves; ~22 seconds tail. That's the normal 'processing' UX window.

Dependency health

S3: healthy (handling 8× baseline uploads fine)
Kafka: healthy but consumer lag climbing
Transcode worker fleet: 100% CPU (as expected — trying to consume as fast as possible)
CDN: 3× normal egress, healthy
App tier: healthy (upload path is fine; only 'processing' UX is affected)
Postgres: healthy
User-visible feed freshness: DEGRADED for viral event videos

Your investigation

You look at the dashboard. Which single decision determines your mitigation strategy?

Hint: Videos are queued. Workers are saturated. What are the levers to reduce backlog time?

60 seconds to decide. What do you do RIGHT NOW?

Hint: You cannot spin up transcode fleet instantly — that takes 5-10 min. You CAN change transcode quality settings via config immediately.

Fleet scales up + quality degrades — queue drains to baseline within 8 min. Users report seeing their videos. What is the postmortem root-cause and top action items?

Hint: The proximate cause was viral upload rate. What's the systemic pattern that made this a crisis?

In the retro, someone asks 'the sports final was predictable — why didn't we pre-scale?' What's the honest answer?

Hint: What does 'predictable' mean in an engineering-planning context?

The team asks 'should we buy Mux/Cloudinary instead of running our own transcode fleet?' What's the honest tradeoff?

Hint: Consider cost at scale + operational-simplicity tradeoff.

Knowledge graph

Learn these first

Media pipeline architecture (upload → transcode → CDN)
S3 direct upload via presigned URLs
Async transcoding with async worker fleets + autoscaling
ML ranking basics (candidate generation + engagement prediction)
Multi-CDN economics and egress-fee negotiation

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.