MASTERCLASS

Gold-standard deep dive

WhatsApp — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001

Architecture Decision Record

Signal Protocol end-to-end encryption (double-ratchet + X3DH) with server-side message storage as opaque bytes (vs. server-side encryption, plaintext-over-TLS, or client-only Bluetooth/mesh)

Context

We are building a global consumer messaging product with a phone-number identity model. Users expect: messages arrive within seconds; a lost/stolen phone doesn't leak the message history; the messaging company cannot read message contents (privacy-first brand); works on 2G networks in emerging markets. Volume at maturity: 3B users, 100B+ messages/day. Regulatory: some governments demand backdoors — we need a story for why we cannot comply even if we wanted to. We are building on Erlang/OTP for its historical fit with WhatsApp's real-world architecture; alternatives considered are Go and Rust.

Constraints

3B users, 100B+ messages/day at maturity — 1M+ msg/sec sustained peak
Availability: 99.99% globally; message delivery guarantee (exactly-once from sender's perspective, once-per-recipient)
Latency: p99 message delivery < 3 seconds sender-to-recipient (including retries)
Privacy: the messaging company MUST NOT be able to read message contents — regulatory + brand + trust
Emerging-market networks: 2G, occasional 5-minute offline periods; store-and-forward is essential
Regulatory: EU DSA + India ITA + various jurisdictions have competing requirements; we need a defensible technical argument
Storage: encrypted-blob storage is cheaper than plaintext (no indexing needed) but message-recovery becomes difficult
Team: 100 engineers; hiring cryptographers is hard but essential for the crypto team

Options considered

Signal Protocol end-to-end encryption (chosen)

Pros

Cryptographically strong — Signal Protocol is peer-reviewed and adopted by billions (Signal, WhatsApp, Meta Messenger, iMessage all use variants)
Server cannot read messages — regulatory / privacy / brand alignment
Perfect forward secrecy — a compromised device does not expose the message history
Client-key rotation via double-ratchet — key exchange happens per-message, not per-session
The Signal Foundation maintains the reference implementation; battle-tested

Cons

Server-side features (spam filtering, cloud backup, message search) require workarounds — client-side or opaque
Message loss on device loss is real: if user loses phone AND encrypted-backup, messages are gone
Regulatory demands for backdoors cannot be met without breaking E2E — publicly documented
Multi-device sync requires careful key-management (Signal has multi-device now but it's non-trivial)

Cost: Cryptographic library maintenance + hiring cost for a 5-10 person crypto team + client-side complexity. Ongoing.

Server-side encryption (TLS in flight + AES-256 at rest, decrypted server-side)

Pros

Server can index/search messages (spam filter, content moderation, search UX)
Cloud backup is trivial (server has plaintext keys)
Easier to comply with lawful-intercept requests (whether or not you should)
Simpler engineering — no client-side crypto

Cons

Server can read all messages — breaks privacy promise
A server-side breach exposes all message history globally
Regulatory: you become the target of every warrant + subpoena globally
Loses the trust/brand differentiation — 'we're just Facebook Messenger with a different UI'

Cost: Simpler engineering but massive legal/PR/brand cost.

Plaintext + TLS only (no at-rest encryption)

Pros

Simplest engineering
Full server-side features

Cons

Any storage backup is plaintext — a leaked backup exposes all messages
Server-side breach = full message-history exposure
GDPR-level regulatory nightmare

Cost: Simple upfront; catastrophic on the first breach.

Client-only (Bluetooth mesh / no server)

Pros

Zero server-side data — cannot be compelled by any government
No delivery relies on server infrastructure

Cons

Cannot deliver a message to an offline recipient reliably
Bluetooth range is <100m; not usable for global messaging
Cannot support group chats above a few members without a mesh coordinator

Cost: Impractical for a global consumer product; useful as a niche crisis-communication tool (see: FireChat).

Chosen solution

Signal Protocol E2E encryption + encrypted server-side store (option 1)

Why

Signal Protocol is the only option that satisfies the privacy contract we've committed to — 'we cannot read your messages.' Server-side encryption gives us the ability to read messages, which regulators will demand access to, and a single breach exposes everyone globally. Plaintext is a non-starter. Client-only cannot work for global asynchronous delivery. The tradeoffs (harder features like server-side search, harder multi-device sync, cannot comply with government demands to read messages) are the point — they are the guarantees users trust. Multiple large-scale deployments (Signal, WhatsApp, Meta Messenger, iMessage) validate this architecture at billions-of-users scale.

Rejected alternatives (with reasons)

Server-side encryption — breaks privacy contract; single breach = global message exposure
Plaintext + TLS — GDPR-level compliance risk; catastrophic first breach
Client-only mesh — cannot serve global async delivery
Hybrid E2E-with-optional-server-key-escrow — technically feasible but destroys the privacy contract; users will not distinguish 'may be read' from 'is read'

Trade-offs accepted

Accept that server-side features (spam filter, moderation, search) become complex or client-side
Accept that message loss on device+backup loss is real (users understand this UX)
Accept the ongoing cost of maintaining a crypto team + Signal Protocol library integration
Accept the regulatory tension — publish clear public statement that we technically cannot comply with 'read all messages' requests

Consequences

Client apps become large + complex (crypto library + key management)
Multi-device support requires careful key-exchange design (recent addition to Signal Protocol)
Cloud backup becomes user-initiated + user-encrypted (their encryption key, not ours)
Content moderation shifts to client-side reporting + human review (not server-side automated scan)
Bug bounties for crypto vulnerabilities become an ongoing budget item (~$500K/yr for high-value crypto bug bounty)

When would we reverse this decision?

If regulatory landscape mandates lawful-intercept globally, we may need to choose 'exit market' vs 'ship backdoor' — publicly document our stance
If a fatal cryptographic weakness is found in Signal Protocol, migrate to next-generation (already researching post-quantum)
If server-side features become the primary product-market fit (e.g., team collaboration replaces peer messaging), revisit — Slack-like server-visible model may be right for that

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the tech lead at a Series A startup building a messaging app for a specific vertical (say, medical care coordination for chronic-illness patients). Your CEO points at WhatsApp and says 'we need that but for healthcare.' You have 3 engineers, 12 weeks, and a HIPAA compliance obligation from day 1. WhatsApp has 3B users; your target is 10K providers + 100K patients in year 1. The CEO's ambition is real — she wants privacy-first, real-time, cross-platform. She's not thought about the cryptographic engineering that WhatsApp actually does.

Constraints

112-week deadline (aligned with a hospital partnership pilot)
23 backend engineers + 1 mobile engineer; none has done cryptography work
310K providers × 100K patients = ~1B messages/year projected (2.7K/day — small scale)
4HIPAA compliance: strict PHI handling, audit logs, BAA with any vendor
5Budget: total infra < $10K/mo
6Regulatory context: patients must be able to revoke provider access; providers must retain 7 years of records (state law)
7The CEO's 'we need WhatsApp' framing is about UX + privacy, not architecture — she doesn't know Signal Protocol exists
8HIPAA and full E2E are in tension: 7-year record retention requires the SERVER to have access, not just clients

Your question

What architecture do you propose, and how do you navigate the HIPAA-vs-E2E tension? Be specific about encryption model, storage, cost, and the compliance conversation.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT

BGP misconfiguration takes down the whole messaging platform globally (Meta 2021-style incident)

PagerDuty alert at 15:39 UTC — but really every PagerDuty alert is firing. External monitoring shows: our services (message-api.example.com, presence.example.com, cdn.example.com) are all unreachable from the public internet. DNS resolution FAILS for our domains. The internal team is confused — some engineers can reach internal systems, some cannot. External customers are reporting our app is entirely broken. You are on-call. Context: the SRE team was doing routine BGP maintenance 30 minutes ago.

Metrics

external.public_endpoint.reachability: 0% (unreachable from all external monitoring)
internal.datacenter.connectivity: DEGRADED (some internal services reachable; some not)
dns.public_records.resolution: FAILING (our authoritative nameservers appear offline)
app.registered_users_online: 0 (unable to check — control plane is down)
message.delivery_rate: 0 msg/sec (was 1M+ msg/sec)
app.support_tickets: skyrocketing (users have no way to reach us since our support portal is also down)
twitter.mentions_of_our_brand: exponential growth
bgp.route_announcements.recent: our AS-number withdrew ALL routes at 15:35 UTC (5 min before alert)
internal.email.delivery: DEGRADED — some engineers cannot receive email because MX records don't resolve

Logs

15:00 UTC — SRE team starts scheduled BGP maintenance to add a new upstream provider
15:34 UTC — SRE team deploys a BGP config change that has a subtle bug: withdraws ALL routes instead of adding new ones
15:35 UTC — routers globally propagate the withdrawal; our AS is no longer routable from the public internet
15:35 UTC — our authoritative DNS nameservers become unreachable — even our OWN engineers can't resolve our domain names
15:36 UTC — the routine 'roll back the config' mechanism ALSO requires DNS to work — chicken and egg
15:38 UTC — some SREs realize the physical access to the datacenter routers is needed
15:39 UTC — external monitoring alerts fire
15:40+ UTC — attempts to communicate internally are hampered because email + Slack + internal tools depend on our own infrastructure

Traces

The dependency circle that made this worse:
BGP config to fix → deploy requires internal deploy pipeline → deploy pipeline requires DNS → DNS server needs to be reachable → BGP config...
The user-facing impact:
User tries to send message → DNS lookup for message-api.example.com FAILS → app shows 'no internet connection' (misleading — their internet is fine, we are unreachable)
The engineer-facing impact:
Engineer tries to SSH to datacenter routers via VPN → VPN endpoint DNS FAILS → engineer must use out-of-band mechanisms (direct IP, cellular hotspot)
Engineer tries to page ops team → PagerDuty webhook DNS FAILS (some services)
Engineer tries to update status page → status page is hosted at status.example.com → same DNS issue → cannot even communicate outage to users

Dependency health

Public reachability: DOWN globally
DNS resolution: FAILING for our domain
Internal Kafka: healthy (running fine, but nothing can send messages to it)
Internal Postgres: healthy
Internal WebSocket gateways: healthy but idle (no incoming connections)
AWS Route 53 hosted zone: exists but AWS's nameservers cannot reach our authoritative servers for delegated records
Physical datacenter power + hardware: fine
Team communication: DEGRADED — email + Slack partially failing due to dependency on affected DNS

Your investigation

You look at the metrics. What is the ONE fact that explains everything?

Hint: External unreachable + internal partially unreachable + DNS failing. What's the common root cause?

You realize the config-rollback mechanism ALSO depends on DNS. What's your mitigation strategy?

Hint: You need to reach the routers physically. What is the plan?

Physical rollback works — BGP announcements resume, DNS starts resolving, services come back online over ~30 min. What are the top postmortem action items?

Hint: The proximate cause was a BGP config bug. What's the systemic failure that made it so catastrophic?

In the retro, someone says 'this happened to Meta in 2021 — we should have known.' Would that be a fair characterization?

Hint: Public postmortems exist for a reason. What's the correct framing?

How would this change your DNS + BGP + monitoring architecture going forward?

Hint: The key insight is 'no critical component should depend on the thing it monitors.'

Knowledge graph

Learn these first

Signal Protocol basics (X3DH + Double Ratchet)
End-to-end encryption tradeoffs (server capabilities vs. privacy)
BGP fundamentals + AS-level routing
Out-of-band operational access patterns (post-Meta-2021)

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.