WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2022 • Jun 15, 2022

Make Your Data FABulous

Philipp Krenn

Scaling Elasticsearch prematurely can silently destroy your scoring accuracy. Discover how to navigate the Fast, Accurate, and Big trade-offs to fix broken distributed searches.

Pause
Mute Enter Fullscreen
#1 about 5 min

Introduction and the CAP theorem for distributed systems

The CAP theorem dictates that distributed stateful systems can only guarantee two of three primary consistency attributes.

#2 about 1 min

Comparing database ACID consistency to CAP theorem timing

The concept of database transaction consistency differs fundamentally from the timing consistency defined in the CAP theorem.

#3 about 3 min

Explaining CAP theorem trade-offs with deserted island analogy

A simple thought experiment demonstrates how isolated network nodes must choose between remaining available or staying consistent.

#4 about 2 min

Understanding fast, accurate, and big data store trade-offs

Distributed data stores balance compromises between near real-time processing, exact calculation results, and multi-node scalability.

#5 about 2 min

Sharding principles and the history of distributed shards

Splitting indices into distinct shards allows data stores to parallelize workload distribution across multiple hardware nodes.

#6 about 5 min

Terms aggregation inaccuracies caused by skewed data routing

Combining local top-N document counts from separated cluster shards leads to missing metrics when data is unevenly distributed.

#7 about 3 min

Fixing distributed document aggregation counts with shard sizing

Adjusting the payload sizes fetched from remote nodes improves statistical accuracy but sequentially consumes more processing time.

#8 about 4 min

Document text search algorithms and BM25 relevance scoring

Search algorithms mathematically evaluate document relevance using term frequency, inverse document frequency, and analyzed field lengths.

#9 about 3 min

Distributed execution using search engine query-then-fetch mechanics

Coordinating nodes fetch preliminary relevance metrics first to parse combined rankings before retrieving large full-text documents.

#10 about 2 min

Correcting irregular relevance scores with global cluster statistics

Explicitly pre-fetching distributed data frequencies normalizes scoring variations that surface when shards possess disproportionate text token counts.

#11 about 5 min

Single shard index constraints and system performance trade-offs

Routing all records through a single isolated shard maximizes query accuracy at the strict expense of distributed scalability.

Matching moments

5:46 min

Maximizing data throughput via distributed stream sharding

David Leitner · World Congress 2022

1:56 min

Keeping systems straightforward to minimize performance bottlenecks at scale

Josip Stuhli Josip Stuhli · Coffee With Developers

4:30 min

Introducing data management and the shift to streaming

Mary Grygleski Mary Grygleski · LIVE

5:31 min

Applying distributed computing principles to agentic artificial intelligence

Mary Grygleski Mary Grygleski · Europe 2026 Virtual

4:33 min

Evaluating algorithmic trade-offs across distributed system implementations

Alexander Reelsen · LIVE

3:55 min

Core architecture and federated execution of SearchOLAP

Andrey Abramov Andrey Abramov · World Congress 2026 Europe

More from World Congress 2022

What the heck do "declarative" and "reactive" actually mean? thumbnail
30:40
about 4 years ago

What the heck do "declarative" and "reactive" actually mean?

André Kovac

The Design and Evolution of C++ thumbnail
1:01:15
over 4 years ago

The Design and Evolution of C++

Bjarne Stroustrup

But, you're not Facebook thumbnail
31:12
over 4 years ago

But, you're not Facebook

Kristijan Ristovski

Communicate efficiently with Software Architecture Diagrams thumbnail
28:11
over 4 years ago

Communicate efficiently with Software Architecture Diagrams

Simon Lasselsberger

Get Ready for Jakarta EE 10! thumbnail
42:53
over 4 years ago

Get Ready for Jakarta EE 10!

Ivar Grimstad

Keep your code refactorable thumbnail
26:46
over 4 years ago

Keep your code refactorable

Gerrit Stapper

Upcoming sessions on this topic

Open session

How We Cut Our API's p99 Latency from Minutes to Under a Second

  • Deepak Agrawal

    Atlassian

    Principal Software Engineer

Open session

Architecting Trusted AI: Coupling Multi-Agent Systems with dbt Semantic Store for Zero-Hallucination

  • Pushkar Mishra

    J.P. Morgan

    Senior Vice President

Open session

We Deleted Our Observability Stack and Rebuilt It with OTEL: 12 Engineers to 4 at 20K+ clusters

  • Yash Sharma

    DigitalOcean

    Developer Advocate

Open session

Multi-Cloud Isn't a Project. It's a Side Effect.

  • Antonio Mendoza Pérez

    Temporal Technologies

    Staff Developer Success Engineer

  • Gaurav Thadani

    Temporal

    Staff Developer Success Engineer

Open session

React Under Fire: Rendering Performance for Streaming Agent UIs

  • Abhirupa Mitra

    Slack

    Senior Member of Technical Staff

Open session

Thinking in Graphs, Reasoning with Gemini!

  • Bhavesh Bhatt

    N/A

    Lead AI Engineer

Related videos

Distributed search under the hood thumbnail
46:58
LIVE • about 5 years ago

Distributed search under the hood

Alexander Reelsen

Leveraging Real time data in FSIs thumbnail
41:51
LIVE • almost 3 years ago

Leveraging Real time data in FSIs

Tim Faulkes

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases thumbnail
32:46
World Congress 2024 • about 2 years ago

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Wei Hu

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases thumbnail
28:41
World Congress 2025 • about 1 year ago

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Wei Hu

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases thumbnail
30:25
World Congress 2026 North America • 18 days ago

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Wei Hu

How building an industry DBMS differs from building a research one thumbnail
30:34
World Congress 2023 • about 3 years ago

How building an industry DBMS differs from building a research one

Markus Dreseler

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.