Skip to main content
Back to Twitter/X timeline
MASTERCLASS
Gold-standard deep dive

Twitter/X timeline — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001
Architecture Decision Record

Hybrid fanout — push-based for users with <10K followers, pull-based (celebrity read cache) for accounts above that threshold, merged at read time (over pure-push or pure-pull)

Context
We are building a public microblogging product with a home-timeline UX. Follower distribution is Pareto: most users have <500 followers; a small tail (~0.1%) have millions. We are targeting ~100M DAU at maturity. Read:write is ~1000:1 (timeline reads dominate). We have Postgres for durable tweet storage and Redis Cluster for caching. The open question is: how do we generate a user's home timeline?
Constraints
  • 100M DAU at maturity; ~500M tweets/day (Twitter's 2013 peak); ~10B timeline reads/day
  • Read:write ratio ~1000:1 — timeline reads are the dominant workload
  • Follower distribution is Pareto: ~99% of users have <10K followers; ~0.1% have 1M+
  • Timeline latency: p99 < 200 ms; the whole ranking pipeline runs within this budget
  • Availability: 99.99% for reads; 99.9% for writes (write pipeline can queue briefly)
  • One tweet from an account with 200M followers generates 200M timeline entries with pure-push
  • The famous 143,199 TPS peak (2013 Castle-in-the-Sky) is the reference peak we design against
Options considered

Pure push (fanout-on-write): every tweet triggers writes to every follower's timeline cache

Pros
  • Reads are trivial — ZREVRANGE from a per-user sorted set, <10 ms
  • Simple mental model; no read-time merge complexity
  • Works cleanly for users with <10K followers (the 99%)
Cons
  • A tweet from a 200M-follower celebrity produces 200M writes — this is the classic 'fanout amplification' problem
  • Sustained write TPS is unmanageable during high-engagement events (World Cup, election night)
  • Storage cost is enormous — 500M tweets × ~500 followers avg = 250B timeline entries/day, most never read
Cost: At 100M DAU, push-only would require ~50× the Redis cluster we actually run at Twitter scale. Publicly discussed by Twitter engineering.

Pure pull (fanout-on-read): timeline generation is JOIN + SORT at read time

Pros
  • Writes are trivial — one INSERT
  • Storage cost is minimal — only the tweets themselves, no per-user timeline duplicates
Cons
  • Reads are expensive — for a user with 200 followees, we scan ~2000 candidate tweets per read
  • 1000:1 read:write ratio means read cost dominates infrastructure
  • Postgres cannot serve 10B timeline reads/day with sub-200ms latency without extreme sharding
  • Ranking (For You) becomes impossible without materializing candidates first
Cost: Compute-cost proportional to read volume × avg followees, dominating budget.

Hybrid (chosen): push for <10K followers, pull for celebrity accounts, merged at read time

Pros
  • Write cost is bounded — a celebrity's tweet is O(1) write to celebrity cache, not O(200M) writes to timelines
  • Read cost is small — pull the pushed timeline + merge with recent celebrity tweets from followees
  • Optimizes exactly at the follower-count cliff where the two costs cross over
  • This is Twitter's actual production pattern, publicly documented in 2013+
Cons
  • Two systems to maintain — push-timeline cache AND celebrity cache
  • Read-time merge adds ~10-20 ms
  • Threshold (10K followers) is a business tuning parameter — cutoff must be revisited as follower distribution changes
  • Celebrity users get slightly stale timelines (celebrity tweets show up at merge, not sync push)
Cost: ~15-20% higher operational cost than pure-push at Twitter scale, but avoids the celebrity-amplification blast radius.

ML-ranked candidates (Meta/Twitter For You style)

Pros
  • Better engagement than reverse-chronological
  • Handles celebrity + regular users uniformly via the ranker
  • Enables recommendations beyond follow graph (trending, relevance)
Cons
  • Requires a large ML infrastructure (feature store + model serving + candidate gen)
  • GPU/silicon cost is dominant — Meta invested in MTIA specifically for this
  • For a small-team startup, this is >12 engineer-months just to reach parity with reverse-chronological
Cost: Only justified above ~50M DAU — below that, hybrid reverse-chronological wins on cost.
Chosen solution

Hybrid push/pull (option 3)

Why
The Pareto distribution of follower counts creates two distinct workloads: regular users (writes are cheap, reads must be fast) and celebrity users (writes are catastrophically expensive, reads are only marginally affected by including them). Pure push handles the first perfectly but breaks on the second. Pure pull is the opposite. Hybrid splits at the exact right cliff (10K followers) where push-cost = pull-cost. This is the canonical Twitter answer publicly discussed in [Twitter's 2013 engineering blog](https://blog.twitter.com/engineering/en_us/a/2013/new-tweets-per-second-record-and-how) and refined multiple times since. ML ranking (option 4) is a legitimate future evolution but adds substantial cost + team dependency before it delivers value.
Rejected alternatives (with reasons)
  • Pure push — celebrity fanout amplification is the specific failure mode that killed the original 2010-era Twitter architecture
  • Pure pull — cannot serve 10B reads/day at <200ms latency without extreme sharding + expensive per-read compute
  • ML-ranked from day 1 — infra cost + team dependency too high for a startup; introduce after ~50M DAU
  • Threshold at 100K followers instead of 10K — moves the crossover point but doesn't change the pattern; 10K is the tuning value from Twitter's own analysis
Trade-offs accepted
  • Accept the operational complexity of two systems (push-timeline cache + celebrity cache)
  • Accept ~10-20 ms of read-time merge latency in exchange for bounded write cost
  • Accept that the 10K threshold is a business dial that must be revisited as follower distribution shifts
  • Accept that celebrity users see their own tweets in a slightly delayed way — merge, not sync push
Consequences
  • Every write path must classify the author (follower count) before deciding fanout strategy
  • Author-follower-count must be cached for O(1) lookup (not a JOIN at write time)
  • Read path becomes a merge — timeline retrieval logic is now O(pushed) + O(celebrity followees) + O(sort)
  • Celebrity-cache is a first-class system with its own capacity planning
  • The 10K threshold gets its own dashboard + tuning discipline
When would we reverse this decision?
  • If we adopt ML ranking (For You), the hybrid decision partially unwinds — ranker becomes source of truth and push/pull is a candidate-generation detail
  • If follower distribution changes such that celebrity accounts represent >5% of users, revisit the threshold
  • If we adopt an entirely different UX (chronological → topic-based), the whole timeline architecture changes
  • If storage cost of push-timelines becomes unaffordable relative to compute, revisit — pure ML-ranking may be cheaper at very-large scale

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the founding backend engineer at a startup building a professional-network product (think LinkedIn for a specific vertical — say, chefs). The founders want to add a 'feed' — users see updates from their connections. They point at Twitter and Instagram and say 'give us that.' You have 3 engineers, 12 weeks, and 50K existing users with a mean of ~30 connections. The founders' pitch to investors specifically mentions 'Twitter-quality feed.' They have not thought about the difference between 50K users × 30 followees × reverse-chronological vs. 100M users × ranked-by-ML.

Constraints
  • 112-week deadline (aligned with investor demo)
  • 23 backend engineers; none has built a feed system at scale
  • 350K existing users with ~30 connections mean (max ~500)
  • 4Existing stack: Postgres + Redis + Sidekiq on AWS
  • 5Budget: dedicated feed infra < $8K/mo (product is early — bootstrapped)
  • 6Founder ask: 'Twitter-quality feed' (aspirational; they mean 'a feed that shows connections' updates')
  • 7Vertical is narrow (chef network) — no celebrities with 200M followers; max follower count is ~5K
  • 8Regulatory: PII care (some content is private to connections only)
Your question

What architecture do you propose, and how do you honestly translate 'Twitter-quality feed' into a scope that ships in 12 weeks with 3 engineers? Be specific about the pull vs push decision, cost, and the founder conversation.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT
Fanout write-storm brings the write path to its knees when a celebrity posts during a viral event

PagerDuty alert at 15:47 UTC. Kafka consumer lag on the fanout topic has spiked from 200ms to 42 seconds within the last 3 minutes. Timeline write latency p99 is now 8.4 seconds (baseline 90ms). Redis cluster CPU is at 92% on three specific shards. Users are reporting 'my timeline is 30 minutes behind.' You are on-call. Context: a headline event just happened, and one of the top-3 celebrity accounts posted about it 4 minutes ago.

Metrics
  • kafka.fanout_topic.consumer_lag: 42,000 ms (baseline 200ms)
  • kafka.fanout_topic.production_rate: 380M msg/min (baseline 5M/min) — 76× baseline
  • kafka.fanout_topic.consumption_rate: 6M msg/min (baseline 5M/min) — barely elevated
  • redis.shard_7.cpu: 94% (baseline 15%)
  • redis.shard_12.cpu: 93% (baseline 15%)
  • redis.shard_19.cpu: 92% (baseline 15%)
  • redis.other_shards.cpu: 15-30% (normal)
  • fanout_worker.pool.busy_ratio: 100% (all workers running at capacity)
  • fanout_worker.queue.depth: 340M pending messages
  • timeline.write.p99_latency: 8,400 ms (baseline 90 ms)
  • timeline.read.p99_latency: 12 ms (unaffected — reads are from cached timelines)
  • author.celebrity_ratio: 0.08% of tweets in last 5 min came from >10M-follower accounts
Logs
  • 15:43 UTC — normal traffic
  • 15:44 UTC — headline event breaks; tweet volume spikes 40%
  • 15:44 UTC — celebrity account @topaccount (145M followers) posts about the event
  • 15:44 UTC — fanout worker enqueues 145M fanout messages for that ONE tweet
  • 15:44-15:47 — Redis shards 7, 12, 19 are receiving concentrated writes (many @topaccount followers hash to them)
  • 15:47 UTC — Kafka consumer lag alarm fires; PagerDuty pages you
  • 15:48 UTC — you check the dashboard; multiple celebrities are now posting; write storm intensifies
Traces
  • Trace of a normal (non-celebrity) tweet fanout:
  • → author posts tweet (200 followers)
  • → app INSERTs tweet
  • → app enqueues 1 fanout message
  • → fanout worker: SELECT follower_ids WHERE followee_id = author (fast)
  • → worker: ZADD to 200 Redis timeline keys (~1ms per ZADD)
  • → Total worker time: ~200ms
  • Trace of a celebrity tweet fanout (the current problem):
  • → author posts tweet (145M followers)
  • → app INSERTs tweet
  • → app enqueues... 145M fanout messages? Or 1 message that expands to 145M?
  • → (this is the design bug — the current code enqueues 1 message + worker fans out; worker takes ~145,000 seconds = 40 hours)
  • → During that 40 hours, 3 shards are saturated because ~2-3% of @topaccount's followers hash to each shard = ~4M keys
Dependency health
  • Kafka: healthy but consumer lag climbing
  • Redis cluster: 3 shards degraded (CPU pinned), rest healthy
  • Postgres: healthy
  • Fanout worker fleet: 100% CPU (as expected — trying to consume)
  • Timeline READ path: healthy (still fast, reads cached timelines)
  • Timeline WRITE path: DEGRADED (fanout backed up)
  • User-visible feed freshness: DEGRADED
Your investigation
1

You look at the dashboard. Which single metric best explains what's happening?

Hint: The write path is broken but the read path is fine. What's the relationship between the celebrity author + Redis shard concentration?
2

You realize the underlying issue: your code is doing pure-push fanout even for celebrities. What's your hypothesis for why this got missed in design + testing?

Hint: The classic 'we designed for hybrid but shipped pure-push' bug. Where does the check happen?
3

60 seconds to decide a mitigation. What do you do RIGHT NOW?

Hint: You have two levers: (a) stop the celebrity from fanning out, or (b) speed up the worker fleet. Which is faster?
4

Kill-switch works — consumer lag drops back to baseline within 10 min. What's your postmortem root-cause and the top action items?

Hint: The proximate cause was the missing follower-count branch. What's the systemic failure that made it possible to ship?
5

In the retro, someone says 'we should have known this could happen from Twitter's public 2013 write-up.' Would that be a fair characterization?

Hint: What's the difference between 'we should have known' and 'we should have designed for it'?
Knowledge graph

Learn these first

  • Pareto distribution of follower counts (celebrity accounts)
  • Push-vs-pull tradeoffs for read-heavy systems (Twitter 2013 engineering blog)
  • Kafka consumer lag as a critical SLI
  • Zipfian load-testing for tail failure modes

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.