MASTERCLASS

Gold-standard deep dive

Distributed Search Engine — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001

Architecture Decision Record

Sharded inverted-index (Elasticsearch on Lucene) with 30 GB shards, RF=2, dedicated coordinator nodes + adaptive replica selection + hedged requests, 30 GB JVM heap cap (compressed oops), and cross-cluster search for global federation — over managed Algolia, self-hosted Solr, Meilisearch, or Postgres full-text search.

Context

We are building distributed search for a multi-tenant catalog + logs + social feed at 8-20K QPS peak with 500M-2B documents indexed. Corpus doubles annually. Query mix: 60% BM25 keyword, 25% filtered aggregation (facets), 10% vector kNN (semantic), 5% cross-language / synonyms. p99 latency budget: 100ms end-to-end. Reference deployments: Uber logs 8 PB Elasticsearch, Netflix Atlas 40 PB, Instacart 200M products, Slack ~50 PB messages+files. Failure model: hot shards during flash sales, JVM GC pauses causing tail-at-scale amplification, coord OOM under bulk-indexing bursts.

Constraints

500M-2B documents; ~40 KB avg _source; 20-80 TB primary storage (before replication)
8-20K QPS peak (Black Friday: 60K QPS spikes)
p99 latency: <100ms for keyword; <200ms for kNN vector; <500ms for facet-heavy
Fresh-write visibility: 1s for catalog (default refresh); 30s for logs (throughput priority)
Availability: 99.95% (search is user-facing but degradable to cache + featured items on failure)
Multi-region: US-East primary; EU + APAC read-followers (CCR-replicated)
Cost bound: $75K/mo at scale (year-2); linear scale w/ corpus size
Team: 4 engineers, 1 with prior Elasticsearch ops experience — this is the critical constraint on managed-vs-self-hosted

Options considered

Elasticsearch/OpenSearch self-hosted + Lucene + hedged requests + CCR (chosen)

Pros

Full control over shard sizing (target 30 GB — the compressed-oops sweet spot)
Adaptive replica selection + hedged requests reduce p99 by 30-70% (Dean & Barroso Tail-at-Scale)
Ingest pipelines for enrichment (PII redact, language detect, embed generation) run at index-time
Cross-cluster search (CCS) for multi-region federation without full data replication
Vector kNN via HNSW indexes (native since ES 8.0) — same cluster as BM25, single query interface
Battle-tested at hyperscale: Uber 8 PB, Netflix 40 PB, GitHub, Slack all run OSS Elasticsearch/OpenSearch

Cons

Requires 1-2 dedicated ops engineers for shard rebalancing, JVM tuning, heap sizing, upgrades
License uncertainty (Elastic License 2.0 vs OpenSearch/Apache 2.0) — pin to OpenSearch or ES 7.10 for long-term dependency
Cold-shard 'first query' latency: 500ms-2s after node restart until file cache warms
JVM GC pauses cause tail-at-scale amplification even with G1GC/ZGC tuning

Cost: ~$45K/mo compute for 20-node r6i.4xlarge cluster + 2 engineers ($400K/yr fully loaded) → ~$85K/mo all-in.

Algolia (managed SaaS search)

Pros

Zero ops — Algolia handles sharding, replication, scaling
Sub-10ms p99 latency (highly optimized C++ engine)
Excellent DX: instant search UI kits, dashboard analytics

Cons

Cost: $1.50-$2.50 per 1000 search operations → $25K-$40K/mo at 2K QPS avg, scales BADLY past that
At 2B documents Algolia becomes $150K-$300K/mo (records + operations both metered)
Limited flexibility: no custom analyzers beyond built-ins, no vector kNN as of 2024, no arbitrary aggregations
Vendor lock-in: re-indexing 2B docs elsewhere = 2-3 months of work + double-run cost
Data residency: US + EU regions only — no APAC-native (200ms+ RTT from APAC users)

Cost: $25K/mo at 2K QPS; $150K-300K/mo at 2B docs + 20K QPS — non-linear cost trajectory.

Solr (self-hosted, Cloud mode)

Pros

Also Lucene-based — same underlying inverted index and BM25 semantics
Better for facet-heavy workloads (Solr's facet engine outperforms ES for high-cardinality facets)
Fully Apache 2.0 licensed — no ambiguity

Cons

Weaker distributed story than ES — ZooKeeper dependency, shard rebalancing is manual
Smaller community + ecosystem — Kibana/OpenSearch Dashboards has no Solr equivalent at parity
Native vector kNN added late (Solr 9.0, 2022) — HNSW performance still behind ES/OpenSearch

Cost: Similar compute cost to ES; slightly higher ops burden due to smaller ecosystem.

Postgres full-text search (tsvector + GIN)

Pros

Zero new infrastructure — reuse existing Postgres cluster
Transactional consistency — search results reflect writes IMMEDIATELY (no refresh interval)
SQL-native — join search results with any other relational data trivially

Cons

Ceiling: ~1-10M documents on a single Postgres node before GIN rebuilds become impractical
BM25 is not native (Postgres uses simpler ts_rank / ts_rank_cd — weaker relevance)
No sharding for search-specific workload — general Postgres sharding (Citus) helps but is complex
No vector kNN natively (pgvector helps but is separate index; combining full-text + vector is awkward)
Kills OLTP workload if search QPS grows — search becomes 40% of CPU under load

Cost: Marginal on existing Postgres up to ~5M docs; ceiling then forces expensive migration.

Meilisearch (Rust, single-node)

Pros

Excellent DX and typo-tolerance out of the box
Very fast on small-to-medium corpuses (<100M docs, <5K QPS)
Simple to operate — single binary, no JVM tuning

Cons

Not truly distributed until v1.x + Cloud offering — single-node write model in OSS
500M-2B document corpus is beyond current Meilisearch operational envelope
Ranking model is proprietary — hard to tune or reason about
Small ecosystem — no Kibana equivalent, limited monitoring tooling

Cost: Cheap at small scale; not viable at 2B docs.

Chosen solution

Elasticsearch/OpenSearch self-hosted with dedicated coord role + hedged requests + CCR

Why

Elasticsearch/OpenSearch is the ONLY option that scales to 2B documents + 20K QPS + multi-region within a $75K/mo budget while providing vector kNN + BM25 + facets + aggregations in a single query interface. Algolia's per-operation pricing is a hard non-starter above ~2K QPS. Postgres full-text search is right up until ~10M docs then falls off a cliff. Solr is equivalent to ES at this scale but its distributed operations story is materially weaker. Meilisearch is the right call at <100M docs but not at our scale. The critical decision is NOT which engine — it's whether we have the ops maturity for self-hosted Lucene: we do (1 engineer with prior ES experience), so we pick self-hosted. If we didn't have that expertise, the answer flips to Algolia and we accept the cost trajectory.

Rejected alternatives (with reasons)

Algolia — non-linear cost trajectory becomes $150K-$300K/mo at 2B docs + 20K QPS
Solr — weaker distributed operations story; vector kNN added late
Postgres full-text search — 10M-doc ceiling, no BM25, kills OLTP under load
Meilisearch — 100M-doc operational envelope, not distributed at OSS tier

Trade-offs accepted

Accept 1-2 engineer FTE ops overhead — shard sizing, JVM tuning, upgrades, incident response
Accept G1GC/ZGC tuning is recurring work — heap sizes, region size, pause target
Accept cold-cache latency (500ms-2s) after node restart or shard migration
Accept cluster-wide operations (mapping updates, rebalance) can trigger yellow states — plan runbooks
Accept licensing risk (Elastic License 2.0 vs Apache 2.0) — pin to OpenSearch for future-proofing

Consequences

Full control over shard sizing (target 30 GB) and JVM heap (30 GB compressed-oops boundary)
Adaptive replica selection + hedged requests give us Dean & Barroso Tail-at-Scale wins
Ingest pipelines centralize enrichment logic — one place for PII redact, language detect, embedding
Vector kNN + BM25 in the same cluster and query — semantic + keyword search unified
Cross-cluster search enables multi-region reads without full data replication cost
40% cost savings vs Algolia at 2B docs — capital reinvested in ML relevance model training
Dedicated Search SRE role emerges within 12 months — this becomes a first-class discipline

When would we reverse this decision?

Ops overhead exceeds 40% of one FTE for 2+ consecutive quarters → managed offering (Elastic Cloud, AWS OpenSearch Service) becomes worth its 30-50% premium
Corpus growth exceeds 5B docs AND query mix becomes facet-dominated → evaluate Solr or split into per-domain clusters
Semantic search becomes 60%+ of queries → dedicated vector DB (Pinecone, Weaviate, Qdrant) may outperform ES HNSW at that mix
Team drops below 3 engineers → cannot maintain self-hosted; migrate to managed offering or Algolia

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You are engineer #4 at a 3-year-old B2C marketplace startup (250 engineers). CTO drops in Monday: 'Search is our #1 customer complaint. We built it on Postgres full-text 4 years ago; now we have 180M product listings and search takes 3-8 seconds. Board approved $500K annual budget for a search rebuild. Options I'm hearing: (1) buy Algolia for $200K/year, (2) buy Elastic Cloud managed for $180K/year, (3) hire 2 engineers and self-host OpenSearch for ~$220K/year total. Timeline: must ship v1 in 4 months. CTO wants Algolia because he can wire it up in 2 weeks. Which do we pick and how do you defend it to the board?'

Constraints

112M MAU, 180M products indexed, 15K search QPS peak, 4-8K sustained
25.4 TB primary corpus (180M docs × 30 KB avg _source); 15 TB with 2x replication
33 backend engineers on search team; ~40 engineers overall
4v1 must ship in 4 months (Q1 2026) to hit board's roadmap commitment
5$500K/year approved — compute + tooling + prorated engineer salaries
6Existing stack: Postgres 14 (main DB), Rails monolith, Go services, AWS us-east-1 primary
7CTO preference: Algolia (fast to ship, zero ops)
8Growth trajectory: expected 12K sustained QPS in 18 months

Your question

What's your recommendation to the board? Address (a) the real 3-year cost of Algolia at growth trajectory, (b) whether the 4-month timeline is realistic for self-hosted, (c) whether a 3-engineer team can operationally support self-hosted OpenSearch, and (d) present a sequenced strategy that threads the timeline pressure without locking in unfavorable long-term economics.

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT

Coordinator OOM cascade — search unavailable during Black Friday peak

PagerDuty alert at 11:50am ET on Black Friday. Search QPS is at 55K (3.3x normal 17K). Coordinator node ES-COORD-2 just threw OutOfMemoryError and went offline. ES-COORD-1 heap is climbing (27 GB / 30 GB). p99 search latency has spiked from 45ms to 4.2s in 3 minutes. Customer-facing catalog is timing out. Marketing is texting the on-call.

Metrics

search.qps: 55K (baseline 17K — 3.3x Black Friday multiplier)
es.coord2.heap_percent: 100% → OOM at 11:50:00 (was 65% at 11:47)
es.coord1.heap_percent: 90% climbing (was 60% at 11:47)
es.coord3.heap_percent: 75% climbing
search.p99_latency_ms: 4200 (baseline 45)
search.p99_latency_ms.by_query_type.aggregation: 6800 (WORST offender)
search.p99_latency_ms.by_query_type.keyword_bm25: 380
es.data_nodes.cpu_percent: 55% average (NOT saturated — data plane is fine)
es.data_nodes.heap_percent: 45% average (NOT saturated)
es.aggregation.max_buckets_setting: 65535 (cluster default, never adjusted)
elb.5xx_rate: 12% (was 0.02% baseline)
affiliate_dashboard.top_terms_query_qps: 8 (baseline 0.5 — sudden spike from partner integration)

Logs

11:47:12 ES-COORD-2 slow_log: 'query took 2145ms, size=50000, terms aggregation, index=catalog'
11:48:31 ES-COORD-2 slow_log: 'query took 3892ms, size=50000, terms aggregation, index=catalog' (concurrent)
11:49:45 ES-COORD-2 GC log: 'G1 Concurrent Cycle 4234ms' (usually <200ms — heap pressure)
11:50:03 ES-COORD-2 log: 'OutOfMemoryError: Java heap space during query reduce phase'
11:50:04 ES-COORD-2 log: 'JVM exit signaled by systemd (auto-restart in 90s)'
11:51:12 ES-COORD-1 GC log: 'G1 Concurrent Cycle 2891ms' (climbing heap)
11:52:00 ES-COORD-1 log: 'circuit_breaking_exception: parent circuit breaker tripped, request rejected'
affiliate-api access log: 'partner_id=aff_47 issued 3 queries with size=50000 terms aggregation between 11:47-11:50'

Dependency health

Data plane (12 data nodes): HEALTHY — CPU 55%, heap 45%, disk I/O <60%
Master quorum (3 dedicated master nodes): HEALTHY — cluster state stable
ELB: HEALTHY at LB layer; 5xx spike is from ES cluster returning 503s
Redis (result cache in front of ES): HEALTHY — 95% hit rate on repeat queries
Postgres (source-of-truth for reindex): HEALTHY — no read pressure from search
Affiliate partner API: 3 clients issuing high-cardinality aggregation queries (root cause candidate)

Your investigation

It's 11:50am and PagerDuty is firing. What's the SINGLE metric you check first to distinguish 'cluster overloaded' from 'JVM heap exhaustion on coord' from 'downstream dependency failure'?

Hint: Think about what fails FIRST in each scenario — the metrics diverge in the first 30 seconds.

You've confirmed it's JVM heap exhaustion on coord nodes. Give your top 3 hypotheses for WHY, ranked by likelihood given the metrics + logs.

Hint: Coord nodes accumulate memory during the 'reduce phase' — after N shards return top-K, coord merges. What could inflate this?

Design a 60-second mitigation you can execute WITHOUT rolling deploys. Constraint: search MUST come back online in 90 seconds. What's your action?

Hint: What can you DYNAMICALLY change without config push — cluster settings, per-request limits?

Postmortem the next day. Name 3 action items ranked by impact, with 'expected reduction in incident probability' quantified for each.

Hint: Some are technical, some are process. Prioritize by risk-adjusted impact.

Draft the customer-facing status page update at T+8 minutes (when search recovers first time). Constraint: honest, no jargon, no over-promising.

Hint: Users saw catalog page timeouts. Tell them what happened, what you're doing, and set realistic expectations.

Knowledge graph

Learn these first

Inverted index + BM25 scoring fundamentals
Scatter-gather query execution + coordinator role
Dean & Barroso Tail-at-Scale + hedged requests
JVM heap sizing (30 GB compressed-oops boundary)

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.