MASTERCLASS

Gold-standard deep dive

Metrics Platform — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001

Architecture Decision Record

Prometheus scrapers + M3DB or VictoriaMetrics for long-term storage + Grafana dashboards + cardinality controls per team + histogram data structures (t-digest / HDR) for percentiles + AI-driven anomaly detection alongside static thresholds — over pure Prometheus (no long-term), Thanos, Datadog Metrics, InfluxDB, or Grafana Cloud managed.

Context

We are building the metrics platform at 1M time-series/sec ingest with cardinality controls. Time-series metrics require ~200 bytes per sample × 1M samples/sec × 86400s × 365 = 6.3 PB/year if unrestricted. Cardinality explosion is the #1 failure mode — every unique label combination creates a new time series, and popular high-cardinality labels (user_id, session_id, request_id) can create BILLIONS of series if not controlled. Reference deployments: Uber M3DB runs at 300M+ active time series. Netflix Atlas ingests 4M metric samples/sec into custom TSDB. Airbnb migrated from Prometheus + Thanos to VictoriaMetrics for 60% cost reduction + better high-cardinality support.

Constraints

1M time-series/sec ingest baseline, 5M/sec peak
100M active time series steady-state, 1B during cardinality-explosion events
Query SLA: dashboard queries < 2 seconds p99, alert evaluation < 500ms
Retention: 30d hot + 90d warm + 1y cold + regulatory (financial metrics 7y)
Cardinality controls: 100K series max per team, 10K per label combination
AI anomaly detection required — static thresholds miss 40% of real anomalies + false-positive 40%
Team: 25 engineers total, 6 on Observability team
Cost bound: $40K/mo infrastructure budget for metrics platform

Options considered

Prometheus scrapers + M3DB (or VictoriaMetrics) + Grafana + cardinality controls (chosen)

Pros

M3DB / VictoriaMetrics purpose-built for high cardinality (100M+ active series)
Prometheus scrapers standard across all services
PromQL query language familiar to team
Histogram data structures for accurate percentiles
Cardinality controls enforced at ingest time
Uber M3DB + Airbnb VictoriaMetrics actual production pattern

Cons

3 stateful surfaces (Prometheus + M3DB + Postgres for Grafana)
M3DB ops complexity (ZooKeeper coordination, cluster management)
Different query language for long-term (M3QL) vs Prometheus (PromQL) — some queries don't translate
12-18 month learning curve for team on M3DB internals

Cost: $25K/mo (M3DB cluster + Prometheus scrapers + Grafana + storage) + $2M/yr Observability team

Pure Prometheus (no long-term storage)

Pros

Simplest architecture — one system
Team knows Prometheus
Standard tool in industry

Cons

Local storage limited to ~30 days retention
No cross-cluster query (federation is bandaid)
SPOF at Prometheus instance level
Cardinality explosion breaks Prometheus (OOM on 10M+ series)

Cost: $3K/mo but no long-term + no HA

Thanos for long-term + Prometheus

Pros

Long-term storage on S3
Global query view across regional Prometheuses
Deduplication of overlapping data
Grafana Labs supported

Cons

S3 query latency higher (2-10 seconds)
Thanos ops complexity — 3 components (sidecar, store, query)
Query performance degrades sharply above 20M active series
Doesn't solve cardinality explosion at ingest

Cost: $12K/mo but ceilinged at ~50M series

Datadog Metrics (managed)

Pros

Zero ops burden
Integrated with logs + traces (unified observability)
Best-in-class UI
AI anomaly detection built-in

Cons

$18 per host/mo × 5000 hosts = $90K/mo baseline
Custom metrics: $0.05 per 100 metrics × 100M metrics = expensive at high cardinality
Vendor lock-in on foundational observability
Total cost at our scale: $150-250K/mo

Cost: $150-250K/mo (6x tiered self-hosted)

InfluxDB Enterprise

Pros

Purpose-built for time-series
InfluxQL familiar to team
Good performance on medium cardinality

Cons

Cardinality ceiling ~10M series in InfluxDB 2.x (much lower than M3DB)
InfluxDB 3.x based on IOx (rewrite) — still maturing as of 2024
Vendor concerns after InfluxData layoffs in 2023

Cost: $30K/mo but cardinality ceilinged

Grafana Cloud (managed)

Pros

Zero ops burden
Integrated with Loki (logs) + Tempo (traces)
Grafana Labs runs the same stack we would run ourselves

Cons

Pricing per active-time-series × per-month = $0.02 × 100M active × 12 = $2.4M/yr
Vendor lock-in similar to Datadog
Cardinality overages at 100K series threshold expensive

Cost: $200K/mo at our scale

Chosen solution

Prometheus + M3DB (or VictoriaMetrics) + Grafana + cardinality controls + histograms + AI anomaly detection

Why

The FIVE constraints that drive M3DB/VictoriaMetrics choice: (1) Cardinality — 100M active series is beyond Thanos (~20M ceiling) and pure Prometheus (~10M OOM point). M3DB was purpose-built by Uber for their 300M+ active series problem. VictoriaMetrics is the open-source alternative with similar capabilities. (2) Query performance — M3DB's inverted index for label-based queries is designed for this scale. Thanos's aggregate-then-scan model breaks down. (3) Cost — self-hosted M3DB at $25K/mo vs Datadog at $150-250K/mo (6x) vs Grafana Cloud at $200K/mo (8x). At $500M+ ARR scale, our observability team of 6 engineers is cheaper than vendor lock-in. (4) Long-term retention — 1-year hot + 7-year regulatory cold requires custom lifecycle policies, hard to bolt on to managed services. (5) AI anomaly detection — static thresholds miss 40% of real anomalies; we build custom ML models on top of M3DB (ownership of data model). Datadog's AI anomaly is a black box we can't tune. Uber + Airbnb + Netflix all landed on custom TSDB (M3DB, VictoriaMetrics, or homegrown) for the same reasons at similar scale.

Rejected alternatives (with reasons)

Pure Prometheus — no long-term + no HA + cardinality OOM ceiling
Thanos — 20M active series ceiling (well below our 100M requirement)
Datadog Metrics — $150-250K/mo (6x self-hosted) + vendor lock-in on foundational observability
InfluxDB Enterprise — 10M cardinality ceiling + vendor concerns after 2023 layoffs
Grafana Cloud — $200K/mo at our scale, similar vendor lock-in as Datadog

Trade-offs accepted

Accept M3DB ops complexity — ZooKeeper coordination + cluster management + custom expertise required
Accept 3 stateful surfaces (Prometheus + M3DB + Postgres for Grafana) — dedicated Observability SRE team
Accept some query pattern mismatch between PromQL (Prometheus) and M3QL (M3DB long-term)
Accept 12-18 month team learning curve on M3DB internals
Accept cardinality controls as first-class engineering discipline — 100K series/team limit enforced at ingest
Accept custom AI anomaly detection ML models — ongoing training + tuning
Accept 15-20 engineer Observability + ML team investment

Consequences

Metrics platform handles 100M active time series without breaking — future-proof for 3x growth
Cardinality controls prevent 'user_id label brings down the cluster' incidents
AI anomaly detection catches 30% more real anomalies than static thresholds
Custom-tuned observability becomes competitive differentiator — engineers hire against this stack
Cost trajectory stays flat at $25K/mo through year-3 scale (vs $250K+/mo Datadog)
Observability team owns cost + performance + cardinality trade-offs — first-class accountability
M3DB + VictoriaMetrics expertise becomes valuable engineering skill

When would we reverse this decision?

If team drops below 4 engineers on Observability → managed Grafana Cloud worth 8x cost premium for zero ops
If cardinality drops below 10M active series sustained → simplify back to Thanos (accept regression)
If unified observability strategy (logs + metrics + traces) becomes critical → Datadog vendor lock-in worth premium
If M3DB project abandonment (vendor concerns) → migrate to VictoriaMetrics (same architecture, different maintainer)

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are Observability lead at a $300M ARR SaaS company. VP Engineering says Monday: 'Datadog Metrics is $180K/mo and going up 30% next year. Alternatives: (1) migrate to Prometheus + M3DB (self-hosted, 6-month project), (2) migrate to Grafana Cloud (managed, $220K/mo but bundled with logs+traces), (3) migrate to VictoriaMetrics (self-hosted, cheaper than M3DB), or (4) negotiate Datadog contract with volume commitment. Board Q3 planning next Wednesday.'

Constraints

1$300M ARR SaaS, growing 40%/yr
2Current: Datadog Metrics ($180K/mo) + Datadog Logs ($120K/mo) + Datadog APM ($90K/mo) = $390K/mo total ($4.7M/yr) — bundled
3Team: 5 Observability engineers (of 200 total)
4Metric volume: 30M active time series, 500K/sec ingest
5Growth: 40% YoY = 42M active series year-1, 60M year-2, 84M year-3
6Existing pain: high-cardinality alerts fail Datadog's default 100K series ceiling — 20-30 support tickets/month
7Board risk-tolerance: high for cost savings, LOW for observability blackout during migration

Your question

Recommend to board Wednesday. Show 3-year TCO for each option + cardinality growth math + team feasibility + migration risk. Address VP's implicit fear: 'is a 6-month migration realistic without losing observability during production incidents?'

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT

Cardinality explosion during A/B test rollout — 20M new time series in 4 hours brings down metrics cluster

PagerDuty alert at 2:15pm PT. M3DB cluster memory 96%, ingest lag 45 seconds and growing. Root cause: an engineer deployed A/B test with 'experiment_id' + 'user_id' + 'variant' labels — creating 3M unique label combinations per hour. Metrics platform CANNOT keep up. Observability is degrading during business hours. Downstream: alert rules failing, dashboards timing out, on-call engineers investigating unrelated incidents cannot query metrics.

Metrics

m3db.cluster.memory_percent: 96
m3db.cluster.active_time_series: 128,000,000 (was 100,000,000 baseline)
m3db.cluster.new_series_last_hour: 3,200,000 (was 5,000 baseline = 640x)
m3db.cluster.ingest_lag_seconds: 45 (was 0.5)
prometheus.scrapers.dropped_samples_per_sec: 8,400 (samples too old — dropped)
grafana.dashboard.query_p99_latency_seconds: 12 (was 0.5 = 24x)
alertmanager.rule_evaluation_failures_last_5min: 120 (was 0)
on_call.engineers_blocked_from_query: 4 (investigating unrelated incidents)
ab_test.experiment_id_47f2.deployment_time: 10:30am PT (3h 45min before alert)
affected_dashboards: 45 pct of all dashboards querying labels involving 'experiment_id'

Logs

10:30:14 deployment: 'A/B test experiment_id=47f2 deployed to production'
10:35:00 prometheus-scrapers: 'series count for metric ab_test_conversion: 5,000'
10:40:00 prometheus-scrapers: 'series count for metric ab_test_conversion: 50,000'
10:55:00 prometheus-scrapers: 'series count for metric ab_test_conversion: 200,000'
11:30:00 prometheus-scrapers: 'series count for metric ab_test_conversion: 1,500,000 — exceeds team quota'
12:15:00 m3db-node-3: 'memory pressure warning — new series ingest rate 3M/hour'
13:30:00 alertmanager: 'alert rule evaluation timeout — dashboard query too slow'
14:15:23 pagerduty: 'ALERT: M3DB cluster memory 96 pct'

Dependency health

M3DB cluster: DEGRADED — 96% memory, ingest lag 45s
Prometheus scrapers: DEGRADED — dropping samples
Grafana dashboards: DEGRADED — 12s p99 query latency
Alertmanager: DEGRADED — rule evaluation failing
On-call engineers: BLOCKED — cannot query metrics for unrelated incident investigations
A/B test service: HEALTHY (unaware of the metric explosion it caused)

Your investigation

It's 2:22pm. M3DB memory 96%, engineers blocked from investigating OTHER incidents. What's your SINGLE highest-priority action in the next 5 minutes?

Hint: The A/B test is emitting metrics with high-cardinality labels. The A/B test cannot be stopped instantly (it's serving customer traffic). But the METRICS EMISSION can be stopped independently.

Assume Prometheus scrapers now dropping the offending metric. But 3M orphan series remain in M3DB. Memory still 96%. Grafana dashboards still slow. What's next?

Hint: The 3M orphan series will age out naturally over 30 days (retention). But we need memory relief NOW. What can we do to accelerate memory recovery?

It's 3:15pm. M3DB memory 78%, dashboards recovering, on-call engineers unblocked. But the A/B test is STILL running (with metric emission blocked at scrape layer). Product team wants to know: 'when will we see A/B test conversion data?'

Hint: The problem is the LABEL CARDINALITY (user_id + experiment_id + variant). What labels are actually needed for the A/B test analysis?

Postmortem the next day. Name 3 action items ranked by impact reduction for future cardinality explosion incidents.

Hint: Some are technical, some are process, some are cultural. Prioritize by risk-adjusted impact.

Draft internal communication to engineering leadership at T+3 hours (5:15pm — after crisis resolved but A/B test still down). Constraint: honest about the cause (engineer error), no shaming, focus on process fix.

Hint: Engineering leadership wants to know: what happened, was anyone hurt, what's the fix. Don't blame the individual engineer.

Knowledge graph

Learn these first

Time-series database internals (M3DB, VictoriaMetrics, InfluxDB)
Cardinality explosion as #1 failure mode
PromQL query language + Prometheus scraper pattern
Histogram data structures for accurate percentiles (t-digest, HDR)

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.