MASTERCLASS

Gold-standard deep dive

Distributed Cache — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001

Architecture Decision Record

Managed Redis Cluster (ElastiCache) over self-built Memcached fork or DynamoDB DAX for the L5 cache tier (500K → 1M QPS)

Context

We are a Series C consumer product. Read traffic just crossed 500K QPS on the primary path and is growing 15% per month. p99 DB latency has crept from 12 ms to 40 ms as the working set outgrew the buffer pool. We need a cache tier that removes ~80% of DB reads within 8 weeks, ahead of a large partnership launch. We are a 3-engineer backend team on AWS; none of us has built a distributed cache from scratch. Our budget for the cache tier is $50K/mo — well above current spend but below the point where FinOps intervenes.

Constraints

Team size: 3 backend engineers, all comfortable with Redis but none with Memcached or custom cache infrastructure at scale
Timeline: 8 weeks to launch, non-negotiable (marketing partnership contract)
Budget: $50K/mo max for the cache tier itself
Availability: match the DB tier's 99.99% SLA
Latency: p99 GET < 3 ms LAN once the cache is populated
Growth: designing for 1M QPS steady-state within 12 months
Workload: 500:1 read:write ratio, Zipfian access pattern (top 100 keys carry ~20% of traffic)
Cloud: AWS is our host; must run in the same VPC as the app tier for network cost + latency

Options considered

AWS ElastiCache Redis Cluster (managed)

Chosen

Pros

Zero build effort — the team can stand up a production cluster in a day, tune it in a week
Fully managed failover, patching, snapshots — no on-call burden for the cache infra itself
Native support for consistent hashing across 16,384 slots + hash tags for co-locating related keys
cache.r6g.4xlarge (100 GB RAM, ~500K ops/sec) scales linearly by adding shards
Fits the workload: Redis Cluster's async replication is the exact right consistency guarantee for cache-aside
Ecosystem: `ioredis`, `lettuce`, and `go-redis` all speak the cluster protocol out of the box

Cons

AWS lock-in — migrating off is a 3-6 month project once we have live traffic
You cannot tune the OS or Redis flags below what ElastiCache exposes (some workloads want `io-threads` or lazyfree-lazy-eviction tweaks)
Per-GB cost is 2-3× a self-managed EC2 fleet at equivalent capacity

Cost: 8 primary shards × cache.r6g.4xlarge + 8 async replicas = ~$32K/mo. Room to grow to 20 shards within budget.

Self-managed Memcached fork (Twitter Twemcache / Facebook mcrouter style)

Pros

Best-in-class throughput per node — Twemcache handles ~1.5M ops/sec on a single m5.4xlarge (source: Twitter's 2012 open-source release)
Simplest protocol (text-based), lowest CPU per op
Full OS + config control — every knob is ours
Lowest raw compute cost at large scale

Cons

12+ weeks to build a production-grade deploy pipeline, monitoring, autoscaling, failover — kills the 8-week deadline
Zero team experience — every operational surprise is a first-time incident
No native cluster protocol — clients need router-side or client-side consistent hashing, which we must build
Missing features: no TTL semantics as rich as Redis, no atomic ops (INCR/DECR) for the analytics side-path we already ship

Cost: Compute-only ~$18K/mo at 1M QPS. Total cost including on-call + engineering time: significantly higher than ElastiCache in year 1.

DynamoDB DAX (in-memory accelerator in front of DynamoDB)

Pros

Fully managed, zero cache-config to write
Sub-ms latency for cache hits
Write-through + read-through are built in — no cache-invalidation code

Cons

Requires migrating the source-of-truth from Postgres to DynamoDB — completely out of scope for the 8-week deadline
DAX is bolted to DynamoDB; useless for our current PG + KV-cache pattern
Cluster tops out at 10 nodes; hard to scale past ~500K QPS without splitting keyspaces

Cost: Meaningless in isolation — the DB migration alone is a $500K+ project.

Redis Enterprise / third-party managed

Pros

Data-tiering (Redis Enterprise) can cache 3-4× the RAM at similar cost by keeping cold keys on NVMe
Cross-region replication built in

Cons

Sales-cycle procurement — 4-8 weeks to contract sign, blowing the deadline
New vendor relationship at a moment when the team should be focused on the launch
Marginal capability gains vs. plain ElastiCache at our scale

Cost: ~$40K/mo at equivalent capacity plus enterprise support fees.

Chosen solution

AWS ElastiCache Redis Cluster (managed)

Why

ElastiCache is the only option that satisfies ALL constraints: 8-week ship (achievable in 2 weeks of infra work), $32K/mo well under budget with headroom to grow, team can operate it Day 1, and the Redis Cluster protocol matches the workload's needs. The Memcached fork is a strictly better answer at 10M QPS — but we are at 1M and paying a 40% premium buys us 12 weeks of engineering time we can spend on the differentiator. This is the classic 'buy vs build' where buy is right for the current constraints and the reversal conditions are well-known.

Rejected alternatives (with reasons)

Self-managed Memcached fork — kills the 8-week deadline and requires expertise the team does not have
DAX — requires migrating source-of-truth to DynamoDB, which is a 6-month project
Redis Enterprise / third-party managed — procurement cycle blows the launch date
Skip caching, vertically scale the DB — buys 3 months at 2× DB cost, then hits a hard wall

Trade-offs accepted

Accept AWS lock-in for the cache tier. Reversible over 3-6 months if we move.
Accept a ~40% cost premium over a self-managed fleet at equivalent capacity. Break-even with self-managed is around 5M QPS.
Accept opaque tuning surface — a few Redis flags we would otherwise touch are not exposed.
Accept that hot-key mitigation and hedged reads must live in the client library rather than the cache — the cache tier is generic.

Consequences

The team ships the cache tier in 2 weeks, freeing 6 weeks for the launch feature-work.
Cache tier cost is 5% of the DB tier saving — every DB read avoided costs $0 downstream.
Every future cache-tier design decision (hot keys, L1 client cache, hedged reads) can be prototyped and rolled out incrementally in the client library.
On-call for the cache tier is minimal — the ElastiCache page is boring, which is what we want at Series C.

When would we reverse this decision?

Sustained load exceeds 5M QPS and per-GB cost becomes the dominant infrastructure line item — evaluate self-managed Redis on EC2 or a Twemcache fork
Latency budget tightens below 1 ms p99 — evaluate client-side L1 tier + Redis Cluster combination, or explore an in-process cache with async invalidation
Multi-region requirement lands — evaluate Redis Enterprise CRDBs, or build application-level cross-region invalidation on top of ElastiCache
Team gains distributed-systems expertise AND we are truly cost-constrained — the Memcached fork story becomes attractive again

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the senior backend lead at a Series C startup. Read traffic just crossed 500K QPS. Your database (RDS Postgres) is at 78% CPU during peak and p99 query latency has crept from 12 ms to 40 ms in the last month. You do not have a cache tier yet — the app hits the DB on every read, with a small in-process LRU in front. The CTO wants a cache tier live in 8 weeks, ahead of a major partnership launch. She also mentioned in a hallway conversation that she 'saw a really neat blog post about how Discord builds their own cache and it saved them millions.' You have 3 backend engineers.

Constraints

18-week deadline (partnership launch is a signed contract; cannot slip)
23 backend engineers total (all comfortable with Redis as a client; none has built distributed cache infrastructure)
3$50K/mo max budget for the cache tier itself
4Availability: match the DB's 99.99% SLA
5Growth: designing for 1M QPS steady-state within 12 months
6Workload: heavy Zipfian skew (top 100 keys ~20% of traffic)
7CTO's Discord-built-their-own-cache remark suggests she is open to persuasion either direction
8The team should NOT be pulled off launch features for cache infra work beyond ~2 weeks

Your question

What architecture do you propose, and how do you address the CTO's Discord-inspired ask? Be specific about product choice, deployment shape, cost, timeline, and the pushback conversation.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT

Redis Cluster hit rate collapsed on one shard during a viral event

PagerDuty alert at 4:23 am. Cache hit rate has dropped from a steady-state 96% to 71% over the last 15 minutes. The alert threshold is 90%. Overall API p99 latency has risen from 45 ms to 380 ms. Traffic is roughly 2× normal for this hour but not unprecedented. Error rate is 0.4% (elevated but not saturating). Users on Twitter are complaining that a specific creator's page is slow. You are on-call.

Metrics

app.api.latency.p99: 380 ms (was 45 ms) — spike started 4:08 am
app.api.latency.p50: 42 ms (near normal)
app.request_rate: 620K QPS (peak ~340K normal; ~2× is high but seen before)
app.error_rate: 0.4% (was 0.02%) — errors trace back to timeouts, not crashes
cache.redis.overall_hit_rate: 71% (was 96%) — recovering slightly the last 3 min
cache.redis.shard_5.hit_rate: 12% (was 95%!) — every OTHER shard is normal
cache.redis.shard_5.cpu: 98% (100% pin during bursts)
cache.redis.shard_5.network_out: 9.4 Gbps (line-rate on 10 Gbps NIC)
cache.redis.shard_5.commandstats: GET dominates — 480K GETs/sec on this shard alone
cache.redis.shard_5.slowlog: 12 SET commands > 100 ms — writes are queuing behind reads
cache.redis.other_shards.hit_rate: 95-96% (steady)
db.postgres.replica_pool.cpu: 84% (was 22%) — this is where the misses are landing
db.postgres.replica_pool.p99_query_latency: 180 ms (was 8 ms)

Logs

4:07 am — normal traffic pattern
4:08 am — sudden spike in requests for a specific `creator_id=847293` — a music-festival announcement post went viral
4:09 am — client-side L1 tier hit rate for that key drops (working set churn from the burst)
4:12 am — shard 5 CPU crosses 80%
4:14 am — shard 5 network_out crosses 8 Gbps
4:18 am — some app pods start seeing GET timeouts on shard 5 (500 ms client-side timeout)
4:19 am — timeout retries pile up in the router, effectively DDoS'ing shard 5 further
4:23 am — alert threshold breached (hit rate < 90%)

Traces

Trace of a slow request for the viral creator's page:
→ ALB: 2 ms
→ App pod: check L1 in-process cache
→ L1 MISS (working set turned over)
→ App pod: route via cluster-aware client to shard 5 (based on CRC16 of `creator:847293:*`)
→ Redis GET creator:847293:profile on shard 5
→ Queue depth on shard 5 = 320 requests deep
→ GET completes in 340 ms (was normally 0.6 ms)
→ App pod: assemble response
→ Total: 380 ms (was normally 45 ms)
Contrast trace for an unaffected user:
→ ALB: 2 ms
→ App pod: L1 MISS
→ Redis GET creator:someone_else on shard 2 → 0.8 ms
→ Total: 44 ms (normal)

Dependency health

ALB: healthy
App tier: 87% healthy pods, 13% seeing timeouts routed exclusively to shard 5
Redis shards 0, 1, 2, 3, 4, 6, 7: healthy
Redis shard 5: DEGRADED (CPU 98%, network out at line rate)
PG primary: healthy (writes are unaffected)
PG read replica: degraded (absorbing the cache misses from shard 5)

Your investigation

You look at the dashboard. Which single metric tells you the most useful thing right now?

Hint: The overall hit rate dropped from 96% to 71% — but is that because ALL shards are struggling, or because ONE shard is?

You correlate the shard-5 spike with the viral-post log line at 4:08 am. What's your hypothesis for why exactly ONE shard is affected?

Hint: Redis Cluster hashes keys via CRC16 mod 16384 to place them on shards. What does that mean for one hot creator?

60 seconds to decide a mitigation. What do you do RIGHT NOW?

Hint: You have two levers: (a) offload from shard 5, or (b) protect shard 5 from further load. Which is faster to execute at 4 am?

L1 TTL bump works — shard 5 CPU falls to 40% within 90 seconds and p99 recovers. What is the postmortem root-cause and top action item?

Hint: The proximate cause was the viral post. What's the systemic failure the proximate cause exposed?

In the retro, someone asks 'should we have caught this in staging?' Would we have? Why or why not?

Hint: What did staging look like when we tested this cache tier? What is unique about viral traffic?

Knowledge graph

Learn these first

Redis operational basics + Cluster mode
Consistent hashing (CRC16 mod 16,384 slots)
Cache-aside pattern
Zipfian access patterns and hot-key intuition

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.