Back to Slack
MASTERCLASS
Gold-standard deep diveSlack — Masterclass
Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.
1. Architecture Decision Record
The format working architects use to document a decision so future teams understand context, options, and reversal conditions.
ADR 001
Architecture Decision RecordWebSocket-first real-time delivery with Server-Sent Events (SSE) fallback for enterprise firewall-restricted environments
Context
We are building a team collaboration app with real-time messaging as the core UX. Users expect < 200 ms end-to-end message delivery latency and durable delivery even when a client goes offline. We must support both consumer-grade networks (home WiFi, cellular) and enterprise networks with restrictive corporate firewalls that sometimes forbid WebSocket. We have already picked our storage layer (sharded MySQL + Redis); the open question is the real-time delivery mechanism to the client.
Constraints
- Target: 20M DAU across ~150K workspaces at maturity; MVP targets 10K DAU
- Message delivery latency: p99 < 200 ms end-to-end (send tap → recipient screen)
- Concurrent connections per gateway node: must support 50K+ steady-state
- Client environments: web (all browsers), mobile (iOS + Android), desktop (Electron)
- Enterprise environments: some corporate firewalls block WebSocket (rare but not zero)
- Mobile battery: message push should NOT drain battery via aggressive polling
- Recoverability: if a client goes offline for hours, they must catch up on missed messages without duplicates
Options considered
WebSocket (bidirectional, persistent) with SSE fallback
Pros
- Sub-100 ms server-push latency in the common case
- Bidirectional: client can send heartbeats + typing indicators over the same connection
- Well-supported: every browser + mobile OS has native WebSocket + fallback libraries
- Backpressure-friendly: server can pause sends via TCP window, unlike SSE which is one-way
- SSE fallback covers the ~1-3% of environments where WebSocket fails
Cons
- Stateful gateway servers: each WebSocket connection pins to one gateway node
- Connection storms on gateway restarts require deliberate mitigation (jittered reconnect + rate limiting)
- Load balancers must speak WebSocket (upgrades HTTP 101) — some legacy LBs cannot
Cost: ~$2K/mo per 50K concurrent connections at c5.xlarge gateway sizing. Linear with DAU.
Server-Sent Events (SSE) — server → client push over HTTP
Pros
- Simpler protocol: unidirectional server → client, just HTTP with chunked transfer
- Works through more firewalls than WebSocket (it's just HTTP)
- Automatic reconnection built into browser EventSource API
Cons
- Unidirectional: client-to-server events (typing, seen, heartbeat) require a separate HTTP path
- No native backpressure: server keeps writing, no way to pause based on client TCP buffer
- HTTP chunked can be slow through some proxies that buffer
- Mobile browsers have lower support quality than WebSocket
Cost: Roughly equivalent to WebSocket at scale; the operational simplicity is the actual saving.
Long-polling
Pros
- Works EVERYWHERE — plain HTTP request that hangs until data arrives
- Simplest possible client code — no connection state
Cons
- Every message costs an HTTP round-trip start-of-connection overhead
- At high message rates, poll storms hammer the gateway
- p99 latency floor of ~500ms due to poll cadence + TCP setup
- Battery drain on mobile is severe (constant HTTPS setup)
Cost: Higher — every message triggers a new TCP connection. Scale wall around 100K DAU without CDN help.
Push notifications only (APNs / FCM)
Pros
- Zero connection state on the server
- Free-tier scale: platforms subsidize the delivery cost
- Excellent battery on mobile — OS handles delivery
Cons
- APNs / FCM latency is variable: 100 ms to several seconds
- No delivery ordering guarantees
- No delivery guarantees at all — push notifications are best-effort
- Desktop and web users still need a separate delivery path
Cost: Zero direct cost, but the missing-delivery-guarantee UX cost is unacceptable for the core chat use case.
Chosen solution
WebSocket-first with SSE fallback
Why
WebSocket is the standard for real-time chat because it minimizes latency, supports bidirectional traffic (typing indicators, heartbeats), and integrates with backpressure. The stateful-gateway concern is well-understood — sticky sessions + jittered client reconnect solve it. Long-polling is a scale wall; push-only cannot meet the delivery-guarantee bar; SSE-only loses the bidirectional benefits. The 1-3% of enterprise environments where WebSocket fails deserve the SSE fallback path, but it should be a fallback, not the primary. The industry has converged on this pattern (Slack, Discord, Facebook Messenger, Google Chat) for exactly these reasons.
Rejected alternatives (with reasons)
- Long-polling — HTTP setup overhead becomes the dominant latency + battery cost at scale
- Push notifications only — best-effort delivery is incompatible with the durable-chat contract
- SSE-only — loses backpressure + bidirectional; the trade-off is not worth the small firewall-friendliness gain
- Custom UDP protocol (like WhatsApp's early days) — rejects load-balancer + edge-CDN + browser support
Trade-offs accepted
- Accept stateful gateway servers — the alternative is complexity in the delivery path we do not need
- Accept that WebSocket connection storms require explicit rate-limiting + client-side jittered reconnect
- Accept the 1-3% firewall-failure long-tail — invest in the SSE fallback path, not in re-architecting the primary
- Accept the mobile battery cost of a persistent connection — mitigate with client-side backgrounding logic (drop the socket after ~5 min idle on mobile, reconnect on foreground)
Consequences
- The gateway tier becomes a first-class service with dedicated SRE ownership
- Every mobile client build must include the WebSocket + SSE fallback + push-notification cascade
- Message-delivery latency SLOs become measurable end-to-end (client → gateway → fan-out → gateway → client)
- Client reconnect logic becomes a discipline: jittered exponential backoff is non-negotiable
- Missed-message catch-up becomes a first-class API: on reconnect, the client sends 'last_seen_msg_id' and the server backfills
When would we reverse this decision?
- If our target market becomes 100% mobile (no desktop), consider dropping WebSocket in favor of a mobile-native push-first architecture
- If we hit enterprise adoption where 20%+ of customers have WebSocket-hostile firewalls, invest more heavily in the SSE fallback
- If message rates grow such that gateway CPU cost dominates infra bill, consider SSE (lower per-connection CPU) as the primary
- If we launch a voice/video product, WebRTC displaces this decision entirely for those flows
2. Business constraint exercise
Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.
Business constraint exercise
You are the tech lead at a Series B SaaS company (project-management tool). Your CEO announces that the product needs a 'Slack-like chat' feature to compete with Notion + Asana. The board wants it shipped in 8 weeks. You have 2 backend engineers (yourself + 1 senior) and 1 frontend engineer. Your product currently has ~15K active teams. The CEO adds: 'Slack has like a thousand engineers building their chat. We just need the basics.'
Constraints
- 18-week deadline (board commitment; this is a competitive-feature ship)
- 22 backend + 1 frontend engineer; nobody has built real-time messaging at scale
- 3Existing product: 15K active teams, ~5 users per team = 75K DAU realistic peak
- 4Existing stack: Rails + Postgres + Redis + Sidekiq on AWS
- 5Budget: dedicated chat infra < $15K/mo (the product itself is a $10/user/mo SaaS)
- 6Feature scope: channels, DMs, mentions, message editing, threading (basic)
- 7Feature scope OUT: enterprise workspace management, video, screen share, integrations marketplace
- 8The CEO's 'Slack has a thousand engineers' framing is aspirational — she wants the outcome (users chat inside our product), not a Slack-scale architecture
Your question
What architecture do you propose, and how do you push back on the 'Slack-like' framing? Be specific about protocol choice, deployment shape, cost, timeline, and the pushback conversation.
3. Production incident scenario
You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.
INCIDENT
Gateway reconnect storm after a 60-second network blipPagerDuty alert at 14:12 UTC. WebSocket gateway tier CPU has spiked from 25% to 98% across all nodes over the last 45 seconds. New WebSocket connection attempts are being rejected (ALB 503s). Existing connected users are unaffected but the ~40% of users who briefly disconnected during a 60-second network blip 30 seconds ago are all trying to reconnect simultaneously. Rate of new connection attempts: 180K/sec (baseline: 3K/sec). You are on-call.
Metrics
- gateway.cpu.avg: 98% (was 25%) — spike started 14:11:15
- gateway.connection_open_rate: 180,000/sec (was 3,000/sec) — 60× baseline
- gateway.connection_reject_rate: 40,000/sec (previously 0) — ALB shedding
- gateway.tcp.time_wait: 240K (was 5K) — sockets in TIME_WAIT accumulating
- gateway.memory.used: 78% (was 45%) — new sockets consuming buffers before CPU can process
- alb.5xx_rate: 40,000/sec (was 0)
- redis.pubsub.subscriber_count: down 40% then rebuilding as reconnects succeed
- auth-service.request_rate: 180,000/sec (was 3,000/sec) — every WebSocket auth is a call to auth-service
- auth-service.p99: 3.2 s (was 40 ms) — auth-service is now the bottleneck
- downstream.postgres.connections: 380/400 pool — approaching exhaustion
Logs
- 14:10:15 — normal traffic pattern
- 14:10:16-14:11:16 — 60-second network blip between us-east-1 and gateway region (transit provider issue)
- 14:11:16 — network heals
- 14:11:17 — clients that disconnected (40% of 500K = 200K) begin reconnecting
- 14:11:17 — 200K reconnect attempts arrive within a 5-second window (no jitter in client-side reconnect logic)
- 14:11:22 — gateway pool CPU crosses 80%
- 14:11:32 — ALB starts shedding new connections; existing sessions still healthy
- 14:12:00 — PagerDuty alert fires
- 14:12:12 — you get paged
Traces
- Trace of a reconnect attempt at 14:11:20:
- → Client sends WebSocket upgrade (HTTP 101)
- → ALB routes to a gateway pod
- → Gateway pod validates session cookie
- → Calls auth-service (blocking): expected 40 ms, actual 3.2 s (auth-service is buried under 60× normal traffic)
- → Gateway pod fetches user's workspace membership from Redis: 20 ms
- → Gateway pod subscribes to per-user Redis pub/sub channel: 10 ms
- → Gateway pod sends WebSocket 101 OK to client
- → Total: 3.25 seconds (was normally ~80 ms)
- During this window, the gateway pod is holding an open TCP socket + a thread waiting on auth-service, blocking other reconnects behind it.
Dependency health
- ALB: healthy (but shedding connections due to gateway back-pressure)
- Gateway tier: DEGRADED (CPU pinned, memory rising)
- Auth service: DEGRADED (60× traffic, p99 3.2 s)
- Redis pub/sub: healthy (was 60% loaded, now 40% because subscribers dropped)
- Postgres pool: WARNING (95% utilization; not yet failing)
- Existing WebSocket sessions: healthy (already-connected users are unaffected)
Your investigation
1
You look at the dashboard. Which single metric tells you the most useful thing right now?
Hint: The alert is about gateway CPU. What upstream metric explains the CPU spike?
2
You correlate the 60-second network blip with the reconnect storm. What's your hypothesis for why this is so bad?
Hint: Look at the client-side reconnect logic. What's missing that turned a network blip into a self-DDoS?
3
60 seconds to decide a mitigation. What do you do RIGHT NOW?
Hint: You have two levers: (a) reduce the reconnect rate at the gateway, or (b) fix the client-side jitter bug. Which is faster to execute at 14:13?
4
Rate limiting works — gateway CPU drops to 50% within 90 s and auth-service recovers to 200 ms p99 within 3 min. What is the postmortem root-cause and top action item?
Hint: The proximate cause was the client-side no-jitter reconnect. What's the systemic failure that made this an easy win?
5
In the retro, someone says 'the network blip lasted 60 seconds — it was outside our control.' Would that be a fair characterization?
Hint: The network blip is external. But what happened next was internal to our system. What's the correct framing?
Knowledge graph
Learn these first
- WebSocket protocol + sticky session load balancing
- Cache-aside pattern
- Redis pub/sub + Kafka fan-out
- Client-side thundering-herd mitigation (jittered exponential backoff)
Where this appears in the curriculum
This is the Gold Standard.
Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.