> Markdown version of [/videos/256-distributed-search-under-the-hood](https://www.wearedevelopers.com/videos/256-distributed-search-under-the-hood). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Distributed search under the hood

How do distributed search engines survive network fallacies? Peek under the hood to see how trading exact precision for probabilistic data structures unlocks massive query performance.

- **Speakers:** Alexander Reelsen
- **Event:** WeAreDevelopers LIVE
- **Published:** September 30, 2021
- **Duration:** 46:58
- **URL:** https://www.wearedevelopers.com/videos/256-distributed-search-under-the-hood

## Summary

Transitioning from single-node databases to distributed architectures addresses strict limits on memory, CPU, and reliability, but intrinsically multiplies system complexity. Expanding operational boundaries exposes applications to the known fallacies of distributed computing—such as assuming latency is zero or networks are reliable. To maintain a unified view of the system, clusters require resilient consensus mechanisms. In Elasticsearch, this is achieved through a dedicated master node that handles cluster state mapping and shard allocation via diff-based updates, avoiding "split-brain" scenarios without bottlenecking high-throughput read/write requests.

Running search queries across multiple shards relies on a two-phase execution strategy: querying the top 'k' hits from relevant shards, merging the lists, and subsequently fetching the actual documents. To prevent hot nodes, routing utilizes adaptive replica selection, analyzing queue limits and response times to balance network traffic. Furthermore, engineers can dramatically optimize query velocity by implementing techniques like Block-Max WAND, which skips non-competitive hits during execution. This illustrates a foundational rule of distributed architecture: "on distributed systems, you get nothing for free"—here trading exact total hit counts for massive performance acceleration.

Aggregating data in distributed environments introduces another layer of complexity, as safely merging top-k aggregations across instances can inadvertently drop valid values if network constraints forbid centralizing the raw data. To mitigate this without overwhelming memory, probabilistic data structures are leveraged. By utilizing HyperLogLog++ for cardinality counting and T-Digest for computing percentiles, systems actively trade a minor, predictable degree of accuracy for bounded memory and execution speed. Ultimately, choosing a distributed database requires evaluating these precise architectural trade-offs, and ensuring your client libraries fully abstract the underlying topology away from end developers.

**Keywords:** distributed search architecture, elasticsearch cluster state, fallacies of distributed computing, shard replication mechanisms, consensus algorithms, split-brain recovery, master node election, two-phase search execution, adaptive replica selection, block-max wand optimization, skipping non-competitive hits, probabilistic data structures, hyperloglog++ cardinality, t-digest percentiles, database topology abstraction

## Chapters

1. **Reasons for adopting distributed systems in modern architectures** (00:02) — Scaling capabilities beyond a single machine enables load balancing, better reliability, and parallel workload execution.
1. **Complexity increases when widening system processing boundaries** (04:38) — Operating across networked machines requires coordination, state synchronization, and handling latency over typical single-node designs.
1. **Addressing the fallacies of distributed computing networks** (07:01) — Assuming reliable networks with zero latency and infinite bandwidth leads to systemic failures during implementations.
1. **Achieving consensus and handling cluster participant failure** (09:22) — Distributed algorithms depend on consensus for leader election, membership tracking, and avoiding destructive split-brain configurations.
1. **Core search and resiliency capabilities of Elasticsearch** (12:14) — Built on Apache Lucene, Elasticsearch processes fast full-text search patterns using deep relevancy scoring and distributed data structures.
1. **Managing cluster state updates through master nodes** (16:47) — Dedicated master nodes track topology and handle structural decisions without blocking regular read or write data operations.
1. **Distributing workload data using primary and replica shards** (21:39) — Splitting active indices into multiple logical shards supports write parallelization while copies prevent immediate data loss.
1. **Executing two-phase distributed search strategies across clusters** (24:13) — Distributed queries utilize a scatter-gather sequence where local node results independently process before retrieving the final document matches.
1. **Selecting nodes intelligently using adaptive replica selection** (26:13) — Tracking past query delays and thread pool capacity ensures optimized node targeting and prevents performance bottlenecks.
1. **Skipping non-competitive search hits in scoring algorithms** (28:02) — Halting execution on documents mathematically unable to reach top ranks speeds up queries at the cost of exact hit totals.
1. **Using probabilistic data structures for distributed aggregations** (30:37) — Calculating cardinality and percentiles efficiently requires embracing approximate accuracy through constant-memory algorithms like HyperLogLog++ and T-Digest.
1. **Evaluating algorithmic trade-offs across distributed system implementations** (36:39) — Studying alternative storage models uncovers how varied database systems prioritize consensus, replication routing, and fault repairs differently.
1. **Audience questions on consensus, hashing, and tokenizers** (41:12) — Technical inquiries cover Elasticsearch coordination formats, targeted murmur hashing for data placement, and custom Unicode tokenization rules.

## Related Moments

- [Core architecture and federated execution of SearchOLAP](https://www.wearedevelopers.com/videos/100212-olap-for-ai-applications-and-why-you-should-care) (from "OLAP for AI Applications and why you should care")
- [Core OpenSearch cluster architecture and terminology](https://www.wearedevelopers.com/videos/490-search-and-aggregations-made-easy-with-opensearch-and-nodejs) (from "Search and aggregations made easy with OpenSearch and NodeJS")
- [Keeping systems straightforward to minimize performance bottlenecks at scale](https://www.wearedevelopers.com/videos/1817-how-to-avoid-tech-hype-traps-josip-stuhli) (from "How to Avoid Tech Hype Traps - Josip Stuhli")
- [Q&A on database vendor lock-in and alternative architectural choices](https://www.wearedevelopers.com/videos/746-swapping-low-latency-data-storage-under-high-load) (from "Swapping Low Latency Data Storage Under High Load")
- [Understanding fast, accurate, and big data store trade-offs](https://www.wearedevelopers.com/videos/430-make-your-data-fabulous) (from "Make Your Data FABulous")
- [Design principles for optimizing distributed application structures](https://www.wearedevelopers.com/videos/1520-fault-tolerance-and-consistency-at-scale-harnessing-the-power-of-distributed-sql-databases) (from "Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases")

## Related Articles

- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)](https://www.wearedevelopers.com/magazine/745-why-event-driven-architecture-isn-t-about-speed-and-when-you-actually-need-it)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Dev Digest 139 - Soft and hard queries](https://www.wearedevelopers.com/magazine/487-dev-digest-139-soft-and-hard-queries)

## Related Jobs

- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/319507-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/353953-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/381484-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [Principal Software Engineer, Database Infrastructure](https://www.wearedevelopers.com/jobs/ext/1465908-principal-software-engineer-database-infrastructure) at **GitHub**
- [Principal Software Engineer, Enterprise AI Platform](https://www.wearedevelopers.com/jobs/ext/1467292-principal-software-engineer-enterprise-ai-platform) at **GitHub**