Notification Platform: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Transactional email — SES (~60% of traffic)
Order confirmation email. Orchestrator resolves template, checks rate limit, sends via SES on tenant's dedicated IP.
gRPC or REST call. Include template_id, user_id, params.
Redis GET on suppression list. If user unsubscribed / bounced / marked spam, skip send.
Redis-based token bucket. If tenant over quota, throttle or reject.
Optimize: Local token cache in orchestrator instance (batched Redis sync every 1s). Reduces Redis load 100x.
Fetch template + substitute variables. Handlebars/Mustache rendering.
Optimize: Cache rendered templates for identical params (rare hit but cheap to try).
Round-robin across warmed IPs in tenant's pool.
AWS SES API call. SES handles DKIM signing, sending, bounce processing.
Optimize: Batch API calls when possible (SES SendBulkTemplatedEmail). 5x throughput.
Return message_id + status.
Email send is **~250-600ms p99** — SES external call is dominant. Our orchestration (suppression + rate limit + template) is <50ms. **The rate-limit + suppression checks are non-negotiable** — sending to bad addresses damages sender reputation. Every notification platform MUST have this.
Bottleneck summary
Notification platform latency is **DOMINATED BY EXTERNAL PROVIDERS** (SES 200-1500ms, Twilio 300-2000ms + carrier tail, FCM/APNS 50-500ms + device delivery tail). Our infrastructure (orchestrator + rate limit + suppression + template) is <100ms. **The architectural focus is on RELIABILITY (rate limits, suppression, IP warming) rather than LATENCY** — you can't optimize what you don't control.
Optimization tips (this architecture)
- **Rate limit per tenant**: Token bucket in Redis with local cache. Prevents one tenant DDoSing provider.
- **Suppression list**: Never send to bounced/unsubscribed. Damages sender reputation.
- **Dedicated IP pools per tenant**: Tenant reputation isolation. Spamhaus-safe.
- **IP warming automation**: Gradually increase send volume. Prevents new IPs from being flagged.
- **Fallback across channels**: SMS fails → email. Never rely on single channel for critical.
- **Batch API calls**: SES SendBulkTemplatedEmail = 5x throughput.
- **Template caching**: Rendered templates cached for identical params (rare hit but cheap).
- **Provider circuit breakers**: If SES 5xx rate > 5%, fall back to SparkPost or Mailgun.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.