System Design Case Study

How does a search system index 10B+ messages with near-real-time indexing?

🎯 Design a chat search system that indexes 10B+ messages with <5s indexing latency and returns full-text results in <100ms
Concepts Involved
Elasticsearch Kafka (CDC) Sharding Indexing CDC

How does a search system index 10B+ messages with <5s indexing latency from send to searchable, and return full-text results in <100ms across the entire message corpus?

Scope: Full-text search across message history · not real-time delivery, not file search, not people search. The hard question: how do you make a message searchable within 5 seconds of being sent while maintaining <100ms query latency across 10B documents✗
10B+
indexed messages
growing at ~10B/day
<5s
index latency
send → searchable
<100ms
query P99 latency
full-text + ranking
100%
permission-aware
only show accessible channels

Functional Requirements

What the system must do · full-text search with permission enforcement and near-real-time indexing

Must Have (Core)

1. Full-text search across all messages in a workspace
2. Results are permission-aware · only show messages from channels the user has access to
3. Near-real-time indexing · message searchable within 5 seconds of being sent
4. Relevance ranking · combine text relevance (TF-IDF/BM25) with recency boost
5. Snippet highlighting · show matching text fragments in search results
6. Support filters: by channel, user, date range, has:file, has:link

Out of Scope (this design)

✗ File content search (separate pipeline for PDF/doc indexing)
✗ People/channel search (simple prefix match on metadata, not full-text)
✗ Message delivery (see Slack Real-Time Messaging for the write path)
✗ Analytics queries (OLAP workload, different system)
✗ Semantic/AI search (vector embeddings · future enhancement)

Non-Functional Requirements

Quality constraints for search at 10B+ document scale

PropertyTargetWhy It Matters / Design Impact
Indexing latency<5s from message send to searchableUsers expect to find a message they just sent. Longer lag causes confusion and re-sends.
Query latencyP99 <100ms for full-text searchSearch must feel instant. Slow search drives users to scroll manually instead.
Throughput200K+ msgs/sec indexed, 10K+ queries/sec servedMust keep up with message ingestion rate without falling behind.
Corpus size10B+ documents indexedFull message history must be searchable · not just recent messages.
Availability99.9% for search (degraded = stale results, not outage)Search unavailability is tolerable briefly if results are stale rather than missing entirely.
ConsistencyEventually consistent (~5s lag acceptable)Near-real-time indexing is sufficient · strong consistency would kill throughput.
SecurityPermission-awareNever show messages from channels user can't access. Filter at query time, not index time.

Scale Estimation

Deriving cluster sizing from message volume and query patterns

StepCalculationResult
110B msgs/day · 86400~115K msgs/sec to index
2Avg message 500 bytes → 115K · 500B~57MB/sec ingestion
3ES cluster sizing: 10B docs · 1KB indexed~10TB index storage
4Shards: 10TB · 50GB/shard~200 shards across cluster
5Query load: 10M users · 2 searches/day · 86400~230 queries/sec avg
6Permission filter: avg user in 50 channelsTerms filter with 50 channel_ids per query

Architecture Overview

Kafka consumer indexes messages into Elasticsearch sharded by workspace_id · permission filtering at query time

INDEXING PATH · Message sent → searchable in <5s Message Service persists to Vitess publishes to Kafka Kafka messages topic partition by workspace_id Search Indexer Kafka consumer group batch: 500 docs per bulk request Elasticsearch Cluster index: messages-{workspace_id} refresh_interval: 1s (NRT) Indexing Latency Breakdown Kafka produce + consume: ~50ms Batch accumulation: ~2s ES bulk index + refresh: ~1s Total: ~3-5s QUERY PATH · User searches "project update" → results in <100ms User search query Search API fetch user's channel list build permission filter Elasticsearch Query BM25 + recency boost filter: channel_id IN [user's channels] Results + Highlights top 20 results snippet with tags Query Latency Breakdown Permission lookup (Redis): ~2ms ES query + highlight: ~60ms Network + serialization: ~10ms Total: ~72ms SHARDING · workspace_id routing for data isolation Each workspace gets its own ES index (or index alias with routing). Queries never cross workspace boundaries. Large workspaces: multiple shards per index. Small workspaces: shared index with routing key. Hot/warm/cold tiering by message age.

Key Design Decisions

Balancing indexing speed, query latency, and permission correctness

DecisionChoiceWhyAlternative Rejected
Search EngineElasticsearchPurpose-built for full-text search. BM25 scoring, highlighting, aggregations. Proven at 10B+ doc scale.MySQL FULLTEXT: doesn't scale. Solr: operationally heavier. Meilisearch: not proven at this scale.
Indexing PipelineKafka consumer → ES bulk APIDecouples indexing from message delivery. Batch 500 docs per bulk request for throughput. Consumer lag = indexing delay, not message delay.Synchronous index on write: adds latency to message send path. CDC from DB: more complex, same result.
Shard StrategyRoute by workspace_idQueries never cross workspace boundaries. Data isolation for enterprise. Hot/warm tiering per workspace.Shard by channel_id: cross-channel search requires scatter-gather across all shards.
Permission ModelQuery-time filter (channel_id IN user's channels)Channel membership changes instantly reflected. No re-indexing needed when user joins/leaves channel.Index-time ACL: requires re-indexing all messages when membership changes · too expensive.
RankingBM25 + recency decayText relevance as primary signal, with exponential decay favoring recent messages. Tunable per workspace.Pure recency: ignores relevance. Pure BM25: old messages dominate for common terms.
Cross-reference: The indexing pipeline consumes from the same Kafka topic used by the Slack Real-Time Messaging fan-out workers. Vitess remains the source of truth · Elasticsearch is a derived, eventually-consistent search index.

Resilience & Edge Cases

Search must be eventually consistent with the source of truth (Vitess) and handle permission changes correctly

#Failure / Edge CaseWhat BreaksHow It's Handled
1Indexer consumer lag spikeMessages not searchable for minutes instead of secondsAuto-scale indexer consumers. Alert on lag >10s. Kafka retains messages · no data loss, just delay.
2User removed from channelShould no longer see messages from that channel in searchQuery-time permission filter uses live channel membership. Removal is instant · no re-indexing needed.
3Message edited or deletedSearch returns stale content or deleted messagesEdit/delete events flow through same Kafka topic → indexer updates/deletes ES document. Eventual consistency (~5s).
4ES cluster node failureQueries to affected shards failEach shard has 1 primary + 2 replicas. Replica promoted automatically. Query routes to healthy replica.
5Hot workspace (10M+ messages)Single shard too large, queries slowSplit into time-based indices (messages-W123-2025-01). Query across recent indices first, paginate to older.
Consistency model: Search is eventually consistent with a ~5s lag. This is acceptable because users don't expect to search for a message they sent 2 seconds ago. The source of truth is always Vitess · if ES is corrupted, rebuild the index from the Kafka topic or Vitess directly.

Tech Stack & Tradeoffs

Each component chosen for a specific role in the search pipeline

ComponentTechnologyWhy ThisWhy Not X
Search EngineElasticsearch (Lucene)BM25 scoring, inverted index, highlighting, aggregations. Proven at 10B+ docs. Near-real-time refresh.Solr: same engine, harder ops. Typesense/Meilisearch: not proven at this scale.
Indexing PipelineKafka → Custom ConsumerDecoupled from write path. Batch bulk indexing (500 docs/request). Replay from offset on failure.Logstash: less control over batching/error handling. Direct DB polling: higher latency, more load on Vitess.
Permission CacheRedis (user → channel set)Sub-ms lookup of user's accessible channels. TTL 60s, refreshed on membership change event.Query Vitess on every search: adds 5-10ms per query. Embed in ES doc: re-index on membership change.
Source of TruthVitess/MySQLDurable message store. ES index can be rebuilt from Vitess at any time. Single source of truth.ES as primary: not designed for transactional writes, no strong consistency guarantees.
Query RoutingCustom Search API (Go)Builds ES query with permission filter, handles pagination, caches frequent queries.Direct ES access from client: exposes internal schema, no permission enforcement.
Cross-reference: The source of truth for messages is the same Vitess/MySQL data model used by the real-time messaging system. Search is a read-optimized derived view · never the primary store.

Interview Cheat Sheet

The 5 things an interviewer wants to hear about chat search at scale

? Kafka Consumer → ES Bulk API (NRT Indexing)
Messages flow through Kafka to a dedicated search indexer consumer group. Batch 500 docs per ES bulk request. ES refresh_interval=1s gives near-real-time searchability. Total send-to-searchable latency: ~3-5 seconds.
? Workspace-Scoped Sharding
Shard by workspace_id · queries never cross workspace boundaries. Each workspace's data is co-located for fast queries. Large workspaces get time-based index splits (monthly). Small workspaces share indices with routing keys.
? Query-Time Permission Filtering
On every search, fetch user's channel list from Redis cache, inject as terms filter in ES query. Channel membership changes are instantly reflected · no re-indexing. This is the key insight: permissions at query time, not index time.
? BM25 + Recency Decay Ranking
Combine Elasticsearch's BM25 text relevance with a function_score decay on timestamp. Recent messages get a boost. Result: "meeting notes" returns last week's messages before year-old ones, even if older ones have higher term frequency.
? ES is Derived · Vitess is Source of Truth
Elasticsearch is a read-optimized derived index. If corrupted, rebuild from Kafka replay or Vitess scan. Never write to ES directly from the client. Edit/delete events flow through the same Kafka pipeline to keep ES consistent.
One-liner answer: "Kafka consumer batches messages into Elasticsearch bulk API for near-real-time indexing (~5s), workspace-scoped sharding for data isolation, query-time permission filtering via cached channel membership, BM25 + recency decay ranking, and ES as a derived index with Vitess as the source of truth."