WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Sep 30, 2021

Distributed search under the hood

Alexander Reelsen

How do distributed search engines survive network fallacies? Peek under the hood to see how trading exact precision for probabilistic data structures unlocks massive query performance.

Pause
Mute Enter Fullscreen
#1 about 5 min

Reasons for adopting distributed systems in modern architectures

Scaling capabilities beyond a single machine enables load balancing, better reliability, and parallel workload execution.

#2 about 3 min

Complexity increases when widening system processing boundaries

Operating across networked machines requires coordination, state synchronization, and handling latency over typical single-node designs.

#3 about 3 min

Addressing the fallacies of distributed computing networks

Assuming reliable networks with zero latency and infinite bandwidth leads to systemic failures during implementations.

#4 about 3 min

Achieving consensus and handling cluster participant failure

Distributed algorithms depend on consensus for leader election, membership tracking, and avoiding destructive split-brain configurations.

#5 about 5 min

Core search and resiliency capabilities of Elasticsearch

Built on Apache Lucene, Elasticsearch processes fast full-text search patterns using deep relevancy scoring and distributed data structures.

#6 about 5 min

Managing cluster state updates through master nodes

Dedicated master nodes track topology and handle structural decisions without blocking regular read or write data operations.

#7 about 3 min

Distributing workload data using primary and replica shards

Splitting active indices into multiple logical shards supports write parallelization while copies prevent immediate data loss.

#8 about 2 min

Executing two-phase distributed search strategies across clusters

Distributed queries utilize a scatter-gather sequence where local node results independently process before retrieving the final document matches.

#9 about 2 min

Selecting nodes intelligently using adaptive replica selection

Tracking past query delays and thread pool capacity ensures optimized node targeting and prevents performance bottlenecks.

#10 about 3 min

Skipping non-competitive search hits in scoring algorithms

Halting execution on documents mathematically unable to reach top ranks speeds up queries at the cost of exact hit totals.

#11 about 7 min

Using probabilistic data structures for distributed aggregations

Calculating cardinality and percentiles efficiently requires embracing approximate accuracy through constant-memory algorithms like HyperLogLog++ and T-Digest.

#12 about 5 min

Evaluating algorithmic trade-offs across distributed system implementations

Studying alternative storage models uncovers how varied database systems prioritize consensus, replication routing, and fault repairs differently.

#13 about 6 min

Audience questions on consensus, hashing, and tokenizers

Technical inquiries cover Elasticsearch coordination formats, targeted murmur hashing for data placement, and custom Unicode tokenization rules.

Matching moments

3:55 min

Core architecture and federated execution of SearchOLAP

Andrey Abramov Andrey Abramov · World Congress 2026 Europe

2:19 min

Core OpenSearch cluster architecture and terminology

Olena Kutsenko · World Congress 2022

1:56 min

Keeping systems straightforward to minimize performance bottlenecks at scale

Josip Stuhli Josip Stuhli · Coffee With Developers

3:16 min

Q&A on database vendor lock-in and alternative architectural choices

George Asafev · World Congress 2023

1:32 min

Understanding fast, accurate, and big data store trade-offs

Philipp Krenn · World Congress 2022

1:03 min

Design principles for optimizing distributed application structures

Wei Hu Wei Hu · World Congress 2025

More from WeAreDevelopers LIVE

WeAreDevelopers LIVE - Is CSS Catching Up to Flash? thumbnail
1:03:33
10 days ago

WeAreDevelopers LIVE - Is CSS Catching Up to Flash?

Chris Heilmann, Daniel Cranney, Bramus Van Damme

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along thumbnail
1:21:17
17 days ago

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along

Chris Heilmann, Daniel Cranney, Krzysztof Cieślak

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak thumbnail
4:23
19 days ago

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Open Source Agent Toolkits thumbnail
1:00:38
24 days ago

WeAreDevelopers LIVE - Open Source Agent Toolkits

Chris Heilmann, Daniel Cranney, Eddie Jaoude

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude thumbnail
6:16
26 days ago

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude

Chris Heilmann, Daniel Cranney

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail
5:46
about 1 month ago

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

Related videos

Make Your Data FABulous thumbnail
28:39
World Congress 2022 • over 4 years ago

Make Your Data FABulous

Philipp Krenn

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases thumbnail
32:46
World Congress 2024 • about 2 years ago

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Wei Hu

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases thumbnail
28:41
World Congress 2025 • about 1 year ago

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Wei Hu

Scaling Databases thumbnail
21:09
World Congress 2024 • about 2 years ago

Scaling Databases

Tobias Petry

Add Location-based Searching to Site with ElasticSearch thumbnail
45:13
LIVE • almost 6 years ago

Add Location-based Searching to Site with ElasticSearch

Derek Binkley

Leveraging Real time data in FSIs thumbnail
41:51
LIVE • almost 3 years ago

Leveraging Real time data in FSIs

Tim Faulkes

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.