WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Sep 30, 2021

Distributed search under the hood

Alexander Reelsen

How do distributed search engines survive network fallacies? Peek under the hood to see how trading exact precision for probabilistic data structures unlocks massive query performance.

Pause
Mute Enter Fullscreen
#1 about 5 min

Reasons for adopting distributed systems in modern architectures

Scaling capabilities beyond a single machine enables load balancing, better reliability, and parallel workload execution.

#2 about 3 min

Complexity increases when widening system processing boundaries

Operating across networked machines requires coordination, state synchronization, and handling latency over typical single-node designs.

#3 about 3 min

Addressing the fallacies of distributed computing networks

Assuming reliable networks with zero latency and infinite bandwidth leads to systemic failures during implementations.

#4 about 3 min

Achieving consensus and handling cluster participant failure

Distributed algorithms depend on consensus for leader election, membership tracking, and avoiding destructive split-brain configurations.

#5 about 5 min

Core search and resiliency capabilities of Elasticsearch

Built on Apache Lucene, Elasticsearch processes fast full-text search patterns using deep relevancy scoring and distributed data structures.

#6 about 5 min

Managing cluster state updates through master nodes

Dedicated master nodes track topology and handle structural decisions without blocking regular read or write data operations.

#7 about 3 min

Distributing workload data using primary and replica shards

Splitting active indices into multiple logical shards supports write parallelization while copies prevent immediate data loss.

#8 about 2 min

Executing two-phase distributed search strategies across clusters

Distributed queries utilize a scatter-gather sequence where local node results independently process before retrieving the final document matches.

#9 about 2 min

Selecting nodes intelligently using adaptive replica selection

Tracking past query delays and thread pool capacity ensures optimized node targeting and prevents performance bottlenecks.

#10 about 3 min

Skipping non-competitive search hits in scoring algorithms

Halting execution on documents mathematically unable to reach top ranks speeds up queries at the cost of exact hit totals.

#11 about 7 min

Using probabilistic data structures for distributed aggregations

Calculating cardinality and percentiles efficiently requires embracing approximate accuracy through constant-memory algorithms like HyperLogLog++ and T-Digest.

#12 about 5 min

Evaluating algorithmic trade-offs across distributed system implementations

Studying alternative storage models uncovers how varied database systems prioritize consensus, replication routing, and fault repairs differently.

#13 about 6 min

Audience questions on consensus, hashing, and tokenizers

Technical inquiries cover Elasticsearch coordination formats, targeted murmur hashing for data placement, and custom Unicode tokenization rules.

Matching moments

3:55 min

Core architecture and federated execution of SearchOLAP

Andrey Abramov Andrey Abramov · WWC Europe 2026

2:19 min

Core OpenSearch cluster architecture and terminology

Olena Kutsenko · WWC 2022

1:56 min

Keeping systems straightforward to minimize performance bottlenecks at scale

Josip Stuhli Josip Stuhli · Coffee With Developers

3:16 min

Q&A on database vendor lock-in and alternative architectural choices

George Asafev · WWC 2023

1:32 min

Understanding fast, accurate, and big data store trade-offs

Philipp Krenn · WWC 2022

1:03 min

Design principles for optimizing distributed application structures

Wei Hu Wei Hu · WWC 2025

More from WeAreDevelopers LIVE

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
5 days ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Node and Package Security thumbnail
1:08:30
5 days ago

WeAreDevelopers LIVE - Node and Package Security

Chris Heilmann, Daniel Cranney, Zbyszek Tenerowicz

WeAreDevelopers LIVE - Streaming HTML thumbnail
1:10:26
12 days ago

WeAreDevelopers LIVE - Streaming HTML

Chris Heilmann, Daniel Cranney, Jad Joubran

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe thumbnail
1:02:34
19 days ago

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe

Chris Heilmann, Daniel Cranney, Justin Halsall, Werner Jeschko, Nicole Jeschko, Mirco Tomić

WeAreDevelopers LIVE - Web Performance in an AI World thumbnail
1:07:59
26 days ago

WeAreDevelopers LIVE - Web Performance in an AI World

Chris Heilmann, Daniel Cranney

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko thumbnail
5:55
about 2 months ago

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Wei Hu

Senior Vice President of Research and Development

Wei Hu
Open session

World Congress 2026 North America

Boring Failover: Predictable Region Recovery Across 5,000 Microservices

Sahil Sabharwal, Garvit Kataria

Sahil Sabharwal
Garvit Kataria
Open session

World Congress 2026 North America

Databases in the Agent Era

Monica Sarbu

Founder and CEO of xata.io

Monica Sarbu
Open session

World Congress 2026 North America

KV Cache Is Not About Speed: It's About Surviving Inference Costs

David vonThenen

AI/ML Leader | Keynote Speaker | OSS Engineer & Developer Advocate | Agentic AI, Deep Learning, Production AI | Python, Go, C++

David vonThenen
Open session

World Congress 2026 North America

Run your agents in Kubernetes: Build once, deploy anywhere. But really?

Michal Salanci

Senior Systems Engineer at ESET Cybersecurity

Michal Salanci
Open session

World Congress 2026 North America

No Single Model to Rule Them All: Building Resilient AI Agents Across Open & Closed LLMs

Emmanuel Acheampong

Senior Manager Developer Relations at Crusoe AI

Emmanuel Acheampong

Related videos

Make Your Data FABulous thumbnail
28:39
WWC 2022 • about 4 years ago

Make Your Data FABulous

Philipp Krenn

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases thumbnail
32:46
WWC 2024 • almost 2 years ago

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Wei Hu

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases thumbnail
28:41
WWC 2025 • 12 months ago

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Wei Hu

Scaling Databases thumbnail
21:09
WWC 2024 • almost 2 years ago

Scaling Databases

Tobias Petry

Add Location-based Searching to Site with ElasticSearch thumbnail
45:13
LIVE • over 5 years ago

Add Location-based Searching to Site with ElasticSearch

Derek Binkley

Leveraging Real time data in FSIs thumbnail
41:51
LIVE • almost 3 years ago

Leveraging Real time data in FSIs

Tim Faulkes

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.