Skip to main content
Back to Notification Platform
MASTERCLASS
Gold-standard deep dive

Notification Platform — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001
Architecture Decision Record

Multi-tenant notification orchestrator + per-tenant priority queues + provider abstraction layer (over per-service ad-hoc integrations OR pure serverless dispatch)

Context
We are building a multi-tenant B2B notification platform (think Twilio Notify, SendGrid, OneSignal). Customers integrate via API to send transactional notifications (order confirmations, delivery updates, alerts) via email, SMS, push, and in-app channels. Volume at maturity: 100B+ notifications/month across 10K+ customers. Different tenants have vastly different volumes (some send 100/day, some send 100M/day). Deliverability discipline is a competitive moat: bad practices (unwarmed IPs, over-throttling, missing bounce handling) cause customer accounts to get blacklisted by receivers (Gmail, Yahoo, mobile carriers), destroying the entire platform's reputation.
Constraints
  • 10K+ tenants, 100B+ notifications/month at maturity
  • Per-tenant volumes range 100/day → 100M/day (6 orders of magnitude)
  • Channels: email (transactional + marketing), SMS (US + international), push (iOS APNs + Android FCM), in-app
  • Availability: 99.99% API-tier; deliverability guarantee: 99%+ for transactional
  • Provider abstraction: we integrate with Amazon SES, SendGrid, Mailgun, Twilio, Vonage, Sinch, Firebase Cloud Messaging, Apple Push Notifications
  • Rate limits: providers throttle senders; must respect per-provider + per-account limits or IPs get blacklisted
  • Deliverability discipline: IP warmup (gradually ramp new sender IPs to avoid spam-filter triggers), bounce handling (hard bounces removed from send list, soft bounces retried), unsubscribe compliance (CAN-SPAM + GDPR + TCPA)
  • Regulatory: TCPA for SMS (US), GDPR for EU, CAN-SPAM for email, various country-specific
Options considered

Multi-tenant orchestrator + per-tenant priority queues + provider abstraction (chosen)

Pros
  • Provider abstraction lets us swap providers per-channel per-region without customer-facing API changes
  • Per-tenant queues prevent noisy-neighbor issues — a high-volume tenant can't starve low-volume tenants
  • Priority queues let transactional notifications (order confirmations) beat marketing notifications (weekly digest) under load
  • Centralized deliverability discipline: IP warmup, bounce handling, unsubscribe compliance done ONCE for all customers
  • Rate limiting done centrally — respects provider limits per-account + per-region
Cons
  • Adding a new provider integration is engineering work (each provider has different APIs, webhooks, error codes)
  • Multi-tenant complexity: tenant-isolation bugs can cause cross-contamination (Customer A's notifications sent from Customer B's IPs)
  • Deliverability is our problem — bad practice by ONE tenant can damage reputation for all tenants
Cost: Engineering: ~30 engineer-months. Ops: ~$50K/mo for infrastructure. Total year-1: ~$1.5M.

Per-service ad-hoc integrations (every service integrates directly with SES + Twilio + FCM)

Pros
  • Simplest architecture — no notification tier
  • Each service owns its own deliverability
Cons
  • Duplicated work — every service reinvents IP warmup, bounce handling, unsubscribe
  • No central rate limiting — services can accidentally exceed provider limits + get blacklisted
  • Compliance nightmare — CAN-SPAM/GDPR compliance implementation differs per service
  • No cross-channel intelligence — user unsubscribes from email but still gets SMS
Cost: Low upfront + catastrophic long-term.

Pure serverless dispatch (Lambda per notification)

Pros
  • Zero infrastructure to manage
  • Auto-scales with traffic
Cons
  • Cannot maintain rate-limiter state across Lambda invocations without external store — every invocation queries + updates a shared counter, becoming the bottleneck
  • Provider rate limits are per-account — Lambda instances would need coordination
  • Cannot implement IP warmup (requires stateful long-term reasoning about IP-to-volume ratios)
  • Cold-start latency compounds at scale
Cost: Moderate direct cost but severe operational + deliverability cost.

Buy Twilio Notify / OneSignal / MessageBird (managed multi-channel notification service)

Pros
  • Zero engineering — API-based
  • Provider handles deliverability, compliance, IP warmup
Cons
  • Cost: $0.001-0.005 per notification × 100B/month = $100M-500M/yr at maturity
  • Vendor lock-in on your critical customer-communication path
  • Cannot customize deliverability tuning for your specific customer base
Cost: $100M-500M/yr at scale. Prohibitive.
Chosen solution

Multi-tenant orchestrator + per-tenant priority queues + provider abstraction (option 1)

Why
At 100B+ notifications/month, both 'buy Twilio' and 'ad-hoc per-service' are prohibitive: buy costs $100M+/yr; ad-hoc costs a catastrophe when a single tenant's bad practice contaminates our IPs. The chosen architecture centralizes the hard-to-do-right parts (deliverability discipline, rate limiting, compliance) while providing per-tenant isolation for noisy-neighbor prevention. Provider abstraction gives us bargaining leverage on provider fees (which are our dominant cost) and geo-diversity (route SMS through Twilio in US, Vonage in EU, Sinch in APAC).
Rejected alternatives (with reasons)
  • Per-service ad-hoc — duplicated work + no central rate limiting + catastrophic deliverability risk
  • Pure serverless — cannot maintain rate-limiter state efficiently at scale
  • Buy Twilio Notify — $100-500M/yr prohibitive
  • Skip multi-channel (only email) — SMS + push are core competitive requirements for modern B2B
Trade-offs accepted
  • Accept per-provider integration engineering (each new provider = 4-8 engineer-weeks)
  • Accept the noisy-neighbor complexity — per-tenant queues + priority queuing require careful scheduling
  • Accept centralized deliverability = centralized responsibility (bad-tenant contamination is our problem to prevent)
  • Accept provider-abstraction complexity — different providers report bounces + delivery differently; normalizing is real work
Consequences
  • Deliverability team becomes a first-class discipline (2-3 engineers focused on IP warmup, bounce handling, ISP relationships)
  • Rate limiter becomes a critical service — every notification passes through it
  • Per-tenant SLAs require per-tenant SLI dashboards + separate alerting
  • Compliance becomes a shared responsibility (CAN-SPAM/GDPR/TCPA implementation lives in the platform, not per-tenant)
  • Provider negotiations become a business-development function (annual contract cycles for SendGrid, Twilio, Vonage)
When would we reverse this decision?
  • If SES/Twilio pricing drops 80%+ (unlikely), reconsider buy
  • If our customer base consolidates to a few very-large tenants, per-tenant queues become overkill; simplify to per-region queues
  • If regulatory requires strict per-tenant compliance isolation (some regulated verticals do), simplify tenants into separate deployments
  • If SMS becomes commodity (already trending), provider-abstraction economics shift

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are the tech lead at a Series A SaaS company building a marketplace product for freelancers. Users receive frequent notifications: new job matches, payment confirmations, chat messages, disputes. Your PM asks for 'SendGrid-quality email + Twilio-quality SMS + FCM push notifications' shipped in 12 weeks. You have 3 engineers, 100K active users sending ~10M notifications/month, and no notification-infrastructure experience. Your existing app has an ad-hoc SES integration for password resets that occasionally gets rate-limited.

Constraints
  • 112-week deadline (aligned with new-product launch)
  • 23 backend engineers; none has done notification infrastructure
  • 310M notifications/month expected; 100K DAU
  • 4Existing stack: Rails + Postgres + Redis + Sidekiq on AWS
  • 5Budget: dedicated notification infra < $5K/mo
  • 6Channels needed: email (transactional + marketing), SMS (US only initially), push (iOS + Android)
  • 7PM's ask: 'like SendGrid + Twilio + FCM'
  • 8Existing SES integration is unreliable + has been rate-limited before
  • 9Compliance: CAN-SPAM (email unsubscribe) + basic TCPA for SMS
Your question

What architecture do you propose, and how do you scope 'SendGrid-quality' into 12 weeks with 3 engineers? Be specific about buy vs. build, per-channel choices, and the PM conversation.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT
Amazon SES rate-limits us + our email domain gets blacklisted mid-Black-Friday

PagerDuty alert at 08:12 UTC on Black Friday. Our transactional email delivery rate has dropped from 99.2% to 47%. Amazon SES is returning 'Throttling' errors on ~40% of send attempts. AND: 3 of our 8 dedicated IPs are showing 'blacklisted' status on Spamhaus SBL. Order-confirmation emails aren't reaching customers. Support tickets are pouring in ('did my order go through?'). You are on-call.

Metrics
  • email.delivery_success_rate: 47% (baseline 99.2%)
  • email.ses.throttling_error_rate: 40% (was 0.1%)
  • email.dedicated_ips.blacklist_status: 3 of 8 IPs on Spamhaus SBL
  • email.ses.send_rate_current: 15K/min (SES account limit: 20K/min)
  • email.orders_pending_confirmation: 47,000 users waiting for order emails
  • support.tickets_last_hour: 2,400 (baseline ~50)
  • sales.black_friday_orders: 3× baseline (as expected)
  • email.bounce_rate: 8% (baseline 1.2%) — this is the underlying deliverability problem
  • email.complaint_rate: 0.6% (industry benchmark: keep below 0.1%)
  • email.recent_deployment: new order-confirmation-email design + subject line, deployed 4 hours ago
Logs
  • 07:00 UTC — normal operations
  • 07:30 UTC — Black Friday sales spike begins
  • 08:00 UTC — email.bounce_rate begins climbing (2% → 5%)
  • 08:03 UTC — email.complaint_rate begins climbing (0.15% → 0.4%)
  • 08:07 UTC — SES throttling errors begin appearing
  • 08:10 UTC — 3 IPs show blacklisted status on Spamhaus SBL
  • 08:12 UTC — PagerDuty alert fires (delivery success rate < 90%)
  • 08:15 UTC — customer support Slack begins flooding
Traces
  • Trace of an order-confirmation email attempt:
  • → Order placed by customer at 08:11:23
  • → Notification service enqueues email job
  • → Sidekiq worker picks up job at 08:11:24
  • → Worker calls SES via SDK → Throttling error: 'Sending rate exceeded'
  • → Worker requeues with exponential backoff (30s → 60s → 120s → ...)
  • → 3 minutes later: worker retries → this time succeeds because SES rate limit reset
  • → But email goes to spam folder because our IP is blacklisted
  • → User sees no email in inbox; opens support ticket
  • The cascade:
  • New email design + subject line 'BLACK FRIDAY DEALS INSIDE!' (from 4-hour-ago deploy) triggers spam filters at scale
  • Complaint rate rises → Gmail + Yahoo mark our IPs as suspicious
  • Bounce rate rises → SES throttles us (per SES's automated deliverability protection)
  • Blacklists (Spamhaus, Barracuda) auto-list our IPs based on complaint patterns
  • We enter a spiral: worse deliverability → more complaints (users mark 'this isn't spam' unlikely) → worse deliverability
Dependency health
  • Amazon SES: healthy but throttling us (rate limit protection)
  • Sidekiq: healthy but retry queue growing
  • Postgres: healthy
  • SendGrid backup (unused): healthy
  • Spamhaus SBL: reports our IPs blacklisted
  • Email domain reputation: DEGRADED
  • Customer trust: DEGRADED
Your investigation
1

You look at the metrics. What is the ROOT cause?

Hint: Trace back through the cascade. What was the triggering event?
2

60 seconds to decide a mitigation. What do you do RIGHT NOW?

Hint: You need to stop the cascade. What's the fastest action?
3

Rollback works — email delivery climbs back to 92% within 20 min. What's your postmortem root cause + top action items?

Hint: The proximate cause was content design. What's the systemic gap?
4

In the retro, someone says 'we shouldn't have used dedicated IPs — shared IPs are more resilient.' Would that be a fair analysis?

Hint: There's a tradeoff. What's the right framing?
5

How do we communicate this externally to affected customers?

Hint: Balance transparency + apology + specific fixes.
Knowledge graph

Learn these first

  • Provider abstraction pattern (adapter + router)
  • Deliverability discipline (bounce classification, complaint monitoring, IP warmup)
  • Regulatory compliance (CAN-SPAM, TCPA, GDPR)
  • Multi-tenant SaaS architecture patterns

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.