Payment System: Latency Waterfall
Break down end-to-end latency by hop and percentile. Understand where the p99 tail comes from — DNS, network, cache lookup, database query, serialization. Real requests have wildly different paths depending on cache-hit location.
Latency Waterfall
Break down end-to-end latency by hop (network, application, database, cache) and percentile (p50/p90/p99/p99.9). Real requests have wildly different paths depending on cache-hit location — pick a scenario to see the full waterfall.
Amara Google 2009: every 100ms of latency = 1% revenue lost. Understanding WHERE the tail comes from is the difference between random optimization and targeted engineering.
Checkout initiate — Saga ack (~99% of requests)
User clicks 'Pay'. System validates, reserves inventory (idempotency-keyed), initiates Saga, returns 'processing' state. Payment continues async.
Mobile/desktop RTT + TLS handshake (resumed). Payment forms often over cellular — expect higher tail latency.
Parse cart + card details, validate format, extract idempotency key (client-supplied UUID).
Atomic SETNX prevents duplicate charges. If key already exists, return cached response (idempotent replay).
Optimize: TTL 24h — covers retry storms without polluting Redis. Track hit rate to identify buggy clients replaying too aggressively.
Fire-and-forget publish to Kafka. Saga orchestrator will pick this up and advance the payment state machine.
Optimize: Use acks=all for payment events — data loss is unacceptable. Trade-off vs acks=1 is ~5ms extra latency.
Response returned immediately after Kafka publish. Client sees 'processing' — settlement continues async.
**Checkout initiate is sub-100ms p99** because Saga orchestration means we DON'T wait for settlement. This is the critical UX pattern: acknowledge quickly, complete slowly. **The mistake is blocking checkout on external processor (Stripe/Adyen) — that pushes p99 to 500-2000ms.** Kafka + Saga decouples user-perceived latency from settlement latency.
Bottleneck summary
Payment checkout latency is **BIMODAL BY DESIGN**: fast ack (sub-100ms) + slow settlement (500-2000ms via external processor). Saga orchestration achieves this by decoupling client-facing response from underlying work. **The dominant latency source in settlement is Stripe/Adyen** (200-1500ms) — external dependencies you can't optimize, only fall back. **Idempotency prevents duplicate charges** on retry — every payment system needs Redis SETNX with 24h TTL.
Optimization tips (this architecture)
- **Saga pattern**: NEVER block user response on external processor. Fire event, ack fast, complete async.
- **Idempotency key**: client-supplied UUID + Redis SETNX with 24h TTL. Prevents duplicate charges from retries.
- **Circuit breaker per processor**: Stripe/Adyen circuit breakers trigger fallback to alternate processor after 3 failures.
- **Kafka acks=all**: For payment events, durability trumps latency. Never lose a payment.
- **Sharded ledger**: Route by account_id. Sub-shard queries (single user's history) stay fast; cross-shard aggregates use analytics DB.
- **Retry policy**: Exponential backoff on 5xx from processor. Distinguish retryable (5xx, timeout) vs non-retryable (4xx, insufficient funds).
- **Two-phase UI**: Show 'processing' immediately, transition to 'confirmed' via WebSocket or notification. Never spinner-forever.
- **Reconciliation**: Nightly job compares ledger vs processor records. Alert on discrepancies. External state drifts — plan for it.
Where to go next
Now that you can see where latency comes from, trace how the architecture EVOLVES to handle 10x more traffic. Or dive into the masterclass for the full ADR + business exercise + incident narrative.