MASTERCLASS

Gold-standard deep dive

Distributed Logging System — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001

Architecture Decision Record

Filebeat agents + Kafka buffer + Elasticsearch 20-node cluster for hot indices (7 days) + S3-backed searchable snapshots for warm (30-90 days) + Glacier Deep Archive for cold (90 days-7 years) + tiered index lifecycle policy — over pure Elasticsearch (no tiering), Splunk (managed), Grafana Loki (label-only indexing), or Datadog Logs (managed).

Context

We are building the distributed logging platform for a company at 1M log events/sec baseline, 100 GB/day ingest, 500 TB/year retention. Log volume grows 40%/yr. Reference deployments: Uber runs 8PB of Elasticsearch for logs (2024 published number). Netflix's logging platform ingests 4M events/sec into a custom-tiered system. Airbnb migrated from Elasticsearch to a mix of ES (hot) + S3 (warm) + Glacier (cold) for 60% cost reduction. Cost dominates: at 500 TB/year on Elasticsearch hot storage = $50K+/mo, on tiered storage = $20K/mo. Query patterns: 85% hot (last 24h), 10% warm (7-90 days), 5% cold (compliance queries, quarterly).

Constraints

1M log events/sec baseline, 5M/sec peak (during incidents when everyone logs verbose)
100 GB/day ingest, 500 TB/year retention (7-year regulatory requirement for financial logs)
Query patterns: 85% hot (last 24h), 10% warm (7-90 days), 5% cold (compliance)
Query SLA: hot < 3 seconds p99, warm < 30 seconds p99, cold < 15 minutes p99
Ingest lag SLA: log-to-searchable < 60 seconds p99
Cost bound: $30K/mo infrastructure budget for logging platform
Team: 30 engineers total, 6 on Observability team
Compliance: 7-year retention for financial logs (SOC 2 + PCI-DSS); 90 days for other
Multi-tenant: 200 services logging to same platform, each with own retention + PII scrub requirements

Options considered

Elasticsearch hot + S3 warm + Glacier cold + Kafka buffer + Filebeat (chosen)

Pros

Elasticsearch: fast full-text search on hot data (last 7 days)
S3 searchable snapshots: warm data queryable within 30 seconds (cheaper than hot ES)
Glacier Deep Archive: 7-year retention at $1/TB/mo ($500/mo for 500 TB)
Kafka buffer absorbs 5x spike (5M events/sec) without dropping
Filebeat agents standard across all services
60% cost reduction vs pure Elasticsearch
Airbnb, Netflix (partial), Uber (partial) actual production pattern

Cons

3 storage tiers to operate + lifecycle policies to manage
S3 searchable snapshots have 30-second query latency (vs sub-second for hot ES)
Glacier cold restore is 12+ hours — must plan compliance queries
Kafka + ES + S3 + Glacier = 4 stateful surfaces

Cost: $20K/mo infrastructure (ES cluster + S3 + Glacier + Kafka) + $2M/yr Observability team

Pure Elasticsearch (no tiering)

Pros

Simplest architecture — one storage tier
Sub-second query on all data
Team knows ES ops

Cons

500 TB × $50/TB/mo (hot SSD) = $25K/mo storage alone
10x more expensive than tiered at year-3 scale (2 PB)
No path to 7-year retention economically
Cluster ops complexity grows with data volume

Cost: $50K+/mo at current scale, $200K+/mo at year-3

Splunk (managed enterprise)

Pros

Zero ops burden — Splunk handles everything
Best-in-class search language (SPL)
Enterprise features (compliance reports, RBAC)

Cons

$3-5 per GB ingested × 100 GB/day × 30 days = $30-50K/mo for ingest alone
500 TB retention: $150K+/mo
Vendor lock-in on foundational infrastructure
Splunk pricing model (per-GB ingested) misaligned with cost trajectory

Cost: $150K-250K/mo — 8-15x tiered ES cost

Grafana Loki (label-only indexing)

Pros

10x cheaper than Elasticsearch (indexes only labels, not full text)
S3-backed storage native
Grafana Labs-supported (Grafana Cloud managed option)

Cons

Full-text search not supported (only label-based queries)
Migration from ES = 12-18 months for query pattern retraining
Compliance query pattern often requires full-text ('find all logs mentioning credit-card-number in Q3') — Loki can't do this
Different query language (LogQL) — team retraining cost

Cost: $5K/mo but query pattern mismatch for compliance

Datadog Logs (managed)

Pros

Zero ops burden
Integrated with metrics + traces (unified observability)
Best-in-class UI + collaboration features

Cons

$0.10 per GB ingested + $1.70 per million events queried
100 GB/day × 30 days × $0.10 = $300/mo per GB retention...actual cost varies but $30-80K/mo typical
Retention limited to 90 days on standard plans; longer retention requires cold storage add-on
Vendor lock-in + unified observability strategy locked to Datadog

Cost: $30-80K/mo, 2-4x tiered ES cost

Chosen solution

Elasticsearch hot + S3 warm + Glacier cold + Kafka buffer + Filebeat agents + tiered index lifecycle policy

Why

The FIVE constraints that drive this architecture: (1) 100 GB/day ingest × 500 TB/year retention = cost dominates ALL other concerns at this scale. Splunk is $150K+/mo (8x our budget). Datadog is $30-80K/mo (borderline). Pure ES is $25K/mo growing to $200K+/mo. Tiered ES + S3 + Glacier is $20K/mo. Cost trajectory diverges over 3 years. (2) 85/10/5 query pattern (hot/warm/cold) means we don't need sub-second query on 90% of data — S3 warm at 30-second query latency is FINE for 10% of queries. (3) 7-year retention regulatory requirement makes Glacier Deep Archive ($1/TB/mo) essential — no other option is economically viable for 500 TB × 7 years = 3.5 PB. (4) Full-text search required for compliance ('find all logs mentioning credit-card-number in Q3') eliminates Loki. (5) Kafka buffer is critical — 5x spike during incidents (5M events/sec) cannot be handled by direct-write to ES; Kafka absorbs the spike, ES drains at sustainable rate. Airbnb + Netflix + Uber all landed on this exact pattern for the same reasons. The build-vs-buy analysis: at $200M+ ARR scale, engineering time on Observability team (6 engineers) is cheaper than Splunk/Datadog vendor lock-in.

Rejected alternatives (with reasons)

Pure Elasticsearch (no tiering) — $200K+/mo at year-3, no path to 7-year retention economically
Splunk managed — $150K-250K/mo (8-15x tiered ES cost), vendor lock-in
Grafana Loki — 10x cheaper but no full-text search (compliance query pattern broken)
Datadog Logs — $30-80K/mo (borderline), vendor lock-in on unified observability

Trade-offs accepted

Accept 3 storage tiers to operate + lifecycle policies to manage
Accept 30-second query latency on warm data (vs sub-second on hot)
Accept 12+ hour restore for cold data (must plan compliance queries)
Accept 4 stateful surfaces (Kafka + ES + S3 + Glacier) — dedicated Observability SRE team
Accept Filebeat agent management across all 200 services — automation via Ansible/Chef
Accept Kafka buffer sizing carefully — must absorb 5x spike (5M events/sec × 30 min buffer = 9 GB Kafka storage)
Accept index lifecycle policies as first-class engineering discipline (mistakes cost $$$$ or break compliance)

Consequences

Logging platform cost stays at $20K/mo through year-3 scale (vs $200K+/mo pure ES)
7-year compliance retention economically viable via Glacier ($500/mo for 500 TB)
Kafka buffer becomes reusable primitive (used for other high-throughput ingest)
Index lifecycle policy discipline flows through all data engineering (learned pattern)
Full-text search retained (compliance query pattern preserved)
Observability team owns cost + performance + compliance trade-offs — heightened responsibility
S3 + Glacier ownership becomes cross-team concern (data engineering + observability + compliance)

When would we reverse this decision?

If cost projection exceeds $50K/mo sustained (would mean 5PB+ data) → evaluate Grafana Cloud managed for lower ops
If compliance query pattern shifts to metric-based (aggregate) → migration to Loki becomes attractive
If team drops below 4 engineers on Observability → managed Datadog worth the 2-4x cost premium
If regulatory retention requirement reduces to 1 year (SOC 2 revised) → simpler 2-tier (hot + warm) architecture

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are Observability lead at a $500M ARR fintech. CTO drops in Monday: 'Splunk contract renewal is $1.2M/yr (up 40% from last year). Alternatives: (1) migrate to Grafana Loki (10x cheaper), (2) migrate to Elasticsearch + S3 tiered (5x cheaper), (3) migrate to Datadog Logs (2x cheaper than Splunk), or (4) renew Splunk contract to avoid migration risk. Team of 6 Observability engineers. Board wants a strategy this Friday.'

Constraints

1$500M ARR fintech, PCI-DSS compliant (financial data — 7-year retention required)
2Current: Splunk enterprise, $1.2M/yr renewal (up from $850K last year — 40% increase)
3Team: 6 Observability engineers (own logs + metrics + traces)
4Volume: 500 GB/day, 2 PB total retention (7-year regulatory)
5Query patterns: 80% hot (compliance investigations), 15% warm, 5% cold
6Migration risk: SOC 2 audit in 6 months — cannot fail
7Splunk expertise: 4 of 6 engineers deep in SPL (Splunk Processing Language), rest partial
8Board risk-tolerance: high for cost savings, LOW for compliance failures

Your question

Recommend to board Friday. Show 3-year TCO for each option + compliance risk + team disruption + migration timeline. Address CTO's implicit fear: 'is any migration safe with PCI-DSS audit in 6 months?'

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT

Elasticsearch cluster red state during production incident — log ingestion frozen while services need logs to diagnose the OTHER incident

PagerDuty alert at 3:47pm ET. Elasticsearch cluster status RED — 3 of 20 nodes down (JVM heap exhaustion during log spike from a SEPARATE production incident). Log ingestion queued in Kafka (backing up to 850 GB). Investigators cannot query logs to diagnose the primary incident. On-call: 'we can't investigate the API outage because logging platform is down. Meta-incident.'

Metrics

elasticsearch.cluster.status: RED (was GREEN)
elasticsearch.nodes.up: 17 of 20 (was 20 of 20)
elasticsearch.jvm.heap_used_percent: 96 pct on remaining nodes (baseline 60 pct)
kafka.topic.logs.consumer_lag_bytes: 850,000,000,000 (850 GB backed up)
kafka.topic.logs.consumer_lag_seconds: 1800 (30 min behind real-time)
elasticsearch.ingest.rejected_count_last_5min: 4,200 (was 0 baseline)
primary_incident.api_service.error_rate_percent: 34 (this is what triggered log spike)
primary_incident.duration_minutes: 47 (already in progress when logging platform failed)
investigator_query_rate_logs_platform: 0 per second (was 100 baseline — investigators locked out)

Logs

15:00:03 primary-incident: 'API service starts returning 5xx errors — investigation begins'
15:12:47 log-ingestion: 'log volume from API service 8x baseline (verbose error logging)'
15:22:14 elasticsearch-node-3: 'JVM heap exhaustion, OOM error'
15:22:15 elasticsearch-node-3: 'process crashed, cluster re-shard beginning'
15:25:33 elasticsearch-node-7: 'JVM heap exhaustion propagates from shard rebalancing pressure'
15:28:52 elasticsearch-node-11: 'JVM heap exhaustion'
15:29:00 elasticsearch: 'cluster status RED — 3 primary shards unassigned'
15:33:14 kafka-consumer: 'ingestion frozen — ES cluster rejecting writes'
15:47:23 investigator: 'cannot query logs — investigation stalled'

Dependency health

Elasticsearch: RED — 3 of 20 nodes down, ingestion frozen
Kafka: HEALTHY — buffering logs (850 GB and growing)
Filebeat agents: HEALTHY — still shipping logs to Kafka
S3 warm tier: HEALTHY — historical queries would work if we could query
Primary incident (API service): STILL BROKEN — 47 min into outage
Investigator team: BLOCKED — cannot query logs to diagnose

Your investigation

It's 3:52pm. Elasticsearch cluster red, log ingestion frozen, investigators locked out during ongoing API outage. What's your SINGLE highest-priority action in the next 10 minutes?

Hint: The immediate problem isn't fixing Elasticsearch — it's giving investigators log access DURING the ongoing incident. Where else can they get logs?

Assume investigators unblocked via kafkacat. Now: how do you recover the Elasticsearch cluster (still RED with 3 nodes down + 96% heap on remaining nodes)?

Hint: Adding more nodes will help, but they'll take time to spin up + rebalance shards. What can you do RIGHT NOW to relieve heap pressure on the remaining 17 nodes?

It's 4:47pm. Elasticsearch cluster back to GREEN, Kafka backlog draining, primary API incident (separate issue) also resolved by another team. Marketing asks: 'How do we explain THIS to customers? Their logs from 3-4pm are... where?'

Hint: Logs are all in Kafka + will eventually be in Elasticsearch. They're not lost. But the specific customer question is: 'can I query my API errors from 3-4pm today?'

Postmortem the next day. Name 3 action items ranked by impact reduction for future log-spike-induced ES cluster failures.

Hint: Some are technical, some are architectural, some are process. Prioritize by risk-adjusted impact.

Draft the customer-facing status page update at T+90 minutes (5:17pm — after ES cluster recovered + Kafka backlog draining, but before ALL logs from 3-4pm window are indexed). Constraint: honest about the delay, precise about recovery ETA, no promises we can't keep.

Hint: Status page is more public than the customer email — potential customers researching, media, tech press. Set the technical narrative honestly.

Knowledge graph

Learn these first

Elasticsearch cluster ops + JVM heap tuning
Kafka buffer pattern for high-throughput ingest
Tiered storage index lifecycle policies (S3 searchable snapshots + Glacier)
Cost trajectory analysis (Splunk vs Datadog vs tiered ES)

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.