> Markdown version of [/videos/430-make-your-data-fabulous?t=457](https://www.wearedevelopers.com/videos/430-make-your-data-fabulous?t=457). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Make Your Data FABulous

Scaling Elasticsearch prematurely can silently destroy your scoring accuracy. Discover how to navigate the Fast, Accurate, and Big trade-offs to fix broken distributed searches.

- **Speakers:** Philipp Krenn
- **Event:** World Congress 2022
- **Published:** June 15, 2022
- **Duration:** 28:39
- **URL:** https://www.wearedevelopers.com/videos/430-make-your-data-fabulous

## Summary

Establishing the foundational trade-offs of stateful distributed systems requires understanding the CAP theorem, which dictates balancing Consistency, Availability, and Partition Tolerance. Because network partitions are statistically inevitable in large data centers, engineers must consciously prioritize either consistency or availability during failure states. This conceptual framework extends into modern data engineering through the FAB theory—a model demonstrating that highly distributed systems must constantly balance being Fast (near real-time), Accurate (exact computational precision), and Big (distributed across multiple nodes).

Using Elasticsearch and a dataset of Star Wars metrics, data distribution mechanisms are shown to directly influence aggregation capabilities and search ranking precision. Modern text algorithms like TF-IDF or its successor, BM25, rank text relevance based on term frequency and inverse document frequency (IDF). However, to save network bandwidth, Elasticsearch executes a default query-then-fetch lifecycle that calculates these IDF scores locally per shard rather than globally. When documents are unevenly distributed across a cluster using hashed routing keys, this purely local calculation leads to unexpected scoring variations and dropped results in terms aggregations.

Mitigating these distributed inaccuracies requires tuning database architecture to explicitly handle edge cases. Engineers can increase a query's shard request size to ensure missing top-N terms are counted, effectively trading execution speed for higher analytical accuracy. Applying a distributed frequency search (DFS query-then-fetch) forces the system to calculate global inverse document frequencies before retrieving any records, equalizing scoring across nodes at the severe cost of query performance. Ultimately, keeping data within a single-shard index provides perfect accuracy and optimal speed, serving as the most sensible architectural default until total data volume mandates multi-node scaling.

**Keywords:** cap theorem, distributed database architecture, elasticsearch sharding, network partition tolerance, fab data theory, distributed terms aggregation, search relevance scoring, inverse document frequency, bm25 algorithm, tf-idf ranking, query-then-fetch execution, distributed frequency search, single shard index scaling, document routing keys, system performance benchmarking

## Chapters

1. **Introduction and the CAP theorem for distributed systems** (00:05) — The CAP theorem dictates that distributed stateful systems can only guarantee two of three primary consistency attributes.
1. **Comparing database ACID consistency to CAP theorem timing** (04:37) — The concept of database transaction consistency differs fundamentally from the timing consistency defined in the CAP theorem.
1. **Explaining CAP theorem trade-offs with deserted island analogy** (05:21) — A simple thought experiment demonstrates how isolated network nodes must choose between remaining available or staying consistent.
1. **Understanding fast, accurate, and big data store trade-offs** (07:37) — Distributed data stores balance compromises between near real-time processing, exact calculation results, and multi-node scalability.
1. **Sharding principles and the history of distributed shards** (09:09) — Splitting indices into distinct shards allows data stores to parallelize workload distribution across multiple hardware nodes.
1. **Terms aggregation inaccuracies caused by skewed data routing** (10:46) — Combining local top-N document counts from separated cluster shards leads to missing metrics when data is unevenly distributed.
1. **Fixing distributed document aggregation counts with shard sizing** (14:51) — Adjusting the payload sizes fetched from remote nodes improves statistical accuracy but sequentially consumes more processing time.
1. **Document text search algorithms and BM25 relevance scoring** (17:20) — Search algorithms mathematically evaluate document relevance using term frequency, inverse document frequency, and analyzed field lengths.
1. **Distributed execution using search engine query-then-fetch mechanics** (20:44) — Coordinating nodes fetch preliminary relevance metrics first to parse combined rankings before retrieving large full-text documents.
1. **Correcting irregular relevance scores with global cluster statistics** (23:15) — Explicitly pre-fetching distributed data frequencies normalizes scoring variations that surface when shards possess disproportionate text token counts.
1. **Single shard index constraints and system performance trade-offs** (24:27) — Routing all records through a single isolated shard maximizes query accuracy at the strict expense of distributed scalability.

## Related Moments

- [Maximizing data throughput via distributed stream sharding](https://www.wearedevelopers.com/videos/494-the-rise-of-reactive-microservices) (from "The Rise of Reactive Microservices")
- [Keeping systems straightforward to minimize performance bottlenecks at scale](https://www.wearedevelopers.com/videos/1817-how-to-avoid-tech-hype-traps-josip-stuhli) (from "How to Avoid Tech Hype Traps - Josip Stuhli")
- [Introducing data management and the shift to streaming](https://www.wearedevelopers.com/videos/538-event-messaging-and-streaming-with-apache-pulsar) (from "Event Messaging and Streaming with Apache Pulsar")
- [Evaluating algorithmic trade-offs across distributed system implementations](https://www.wearedevelopers.com/videos/256-distributed-search-under-the-hood) (from "Distributed search under the hood")
- [Exploring query boundaries, data storage, and architecture limits](https://www.wearedevelopers.com/videos/168-kafka-streams-microservices) (from "Kafka Streams Microservices")
- [Core architecture and federated execution of SearchOLAP](https://www.wearedevelopers.com/videos/100212-olap-for-ai-applications-and-why-you-should-care) (from "OLAP for AI Applications and why you should care")

## Related Articles

- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)](https://www.wearedevelopers.com/magazine/745-why-event-driven-architecture-isn-t-about-speed-and-when-you-actually-need-it)
- [Dev Digest 139 - Soft and hard queries](https://www.wearedevelopers.com/magazine/487-dev-digest-139-soft-and-hard-queries)

## Related Jobs

- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/319507-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/353953-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Lead Software Engineer - Data Engineering](https://www.wearedevelopers.com/jobs/ext/2000968-lead-software-engineer-data-engineering) at **Dynatrace**
- [Principal Software Engineer, Database Infrastructure](https://www.wearedevelopers.com/jobs/ext/1465908-principal-software-engineer-database-infrastructure) at **GitHub**
- [Cloud Foundations Team](https://www.wearedevelopers.com/jobs/ext/1483289-cloud-foundations-team) at **GitHub**