> Markdown version of [/videos/1649-let-s-get-aggregated-custom-udafs-in-spark](https://www.wearedevelopers.com/videos/1649-let-s-get-aggregated-custom-udafs-in-spark). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Let's Get Aggregated: Custom UDAFs in Spark  

Stop letting expensive data shuffles bottleneck your Spark workflows. Custom UDAFs pre-aggregate data locally, condensing complex analytics into a single highly performant shuffle.

- **Speakers:** [Nikolai Nikolaev](https://www.wearedevelopers.com/@nikolai-nikolaev)
- **Event:** World Congress 2025
- **Published:** August 20, 2025
- **Duration:** 19:58
- **URL:** https://www.wearedevelopers.com/videos/1649-let-s-get-aggregated-custom-udafs-in-spark

## Summary

Apache Spark’s standard aggregation functions are sufficient for basic data processing, but complex analytical workloads across massive datasets often demand more specialized and performant solutions. Relying strictly on combinations of standard grouping and windowing functions frequently leads to execution plans with multiple expensive data shuffles, creating a major performance bottleneck for distributed engineering workflows.

To drastically minimize these network-heavy operations, data teams can create Custom User-Defined Aggregate Functions (UDAFs). Building a UDAF in Spark involves defining distinct functional stages: `zero` (the initial buffer state), `reduce` (row-by-row partition processing), `merge` (combining intermediate results), and `finish` (finalizing the output), alongside necessary structural encoders like Kryo. This "reduce, merge, finish" pipeline pre-aggregates data on local executors, guaranteeing that data is only shuffled once when combining the intermediate outputs.

Through a practical word-frequency application, custom UDAFs achieve in a single data shuffle what typical ChatGPT-generated Spark code requires two shuffles to perform. Beyond pure optimization, UDAFs provide immense architectural flexibility. They natively support custom data structures like Scala case classes, enforce code reusability when published as shared artifacts, and seamlessly integrate into the Spark SQL API—allowing data scientists and analysts to invoke deeply complex logic using simple native SQL wrappers.

**Keywords:** apache spark aggregation, custom udaf implementation, user-defined aggregate functions, distributed data processing, data shuffling optimization, spark sql api integration, scala implicit classes, kryo encoders, spark execution plan, executor pre-aggregation, reduce and merge operations, window functions performance, big data performance tuning, complex analytical workloads, spark code reusability

## Chapters

1. **Motivations for custom aggregation functions in large datasets** (00:05) — Standard aggregation functions often fall short when processing complex analytical workloads across billion-row datasets.
1. **Defining the core methods of user-defined aggregator functions** (02:14) — Custom aggregations require implementing zero, reduce, merge, and finish functions alongside appropriate encoders.
1. **Understanding distributed execution and the necessity of pre-aggregation** (03:56) — Executing custom aggregations in a distributed framework minimizes network data shuffling by pre-aggregating data locally on executors.
1. **Analyzing baseline performance issues in standard word count techniques** (05:56) — Standard approaches to grouping and windowing functions generate multiple expensive data shuffles within the execution plan.
1. **Designing custom aggregation workflows for efficient frequency mapping** (08:22) — Defining specialized types and mapping logic allows for streamlined reduction and merging phases without repetitive native queries.
1. **Implementing core zero, reduce, merge, and finish functions** (11:48) — Developing each stage involves tracking frequencies in mapped structures and handling null checks within encoders.
1. **Comparing execution plans to benchmark reduced data shuffling** (14:36) — Executing the user-defined aggregation produces an optimized execution plan featuring only a single data shuffle operation.
1. **Leveraging complex data structures in production aggregation workflows** (15:41) — Utilizing case classes as buffers and output values enables calculating multiple metrics reliably within a single operation.
1. **Optimal scenarios for deploying custom defined aggregator functions** (17:13) — Custom functions optimize complex aggregation performance, enable code reusability across teams, and integrate seamlessly with Spark SQL APIs.

## Related Moments

- [Refactoring data science workflows using Rapids QDF and Pandas](https://www.wearedevelopers.com/videos/859-accelerating-python-on-gpus) (from "Accelerating Python on GPUs")
- [Understanding RDDs, DataFrames, and Datasets in Spark](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) (from "PySpark - Combining Machine Learning & Big Data")
- [Comparing offline data analytics with online stream processing](https://www.wearedevelopers.com/videos/944-building-the-platform-for-providing-ml-predictions-based-on-real-time-player-activity) (from "Building the platform for providing ML predictions based on real-time player activity")
- [Pre-calculating data aggregates using event-driven methodologies](https://www.wearedevelopers.com/videos/19-building-high-performance-and-scalable-architectures-for-enterprises) (from "Building high performance and scalable architectures for enterprises")
- [Capabilities of the Apache Spark processing engine](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) (from "PySpark - Combining Machine Learning & Big Data")
- [Summary of decoupling analytical compute and storage](https://www.wearedevelopers.com/videos/100075-parquet-delta-iceberg-ducklake-an-introduction-for-developers) (from "Parquet, Delta, Iceberg & Ducklake - An introduction for developers")

## Related Articles

- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)](https://www.wearedevelopers.com/magazine/745-why-event-driven-architecture-isn-t-about-speed-and-when-you-actually-need-it)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Stephan Gillich - Bringing AI Everywhere](https://www.wearedevelopers.com/magazine/489-stephan-gillich-bringing-ai-everywhere)

## Related Jobs

- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [Software Engineer (L3) Data Substrate](https://www.wearedevelopers.com/jobs/ext/461301-software-engineer-l3-data-substrate) at **Twilio**
- [Software Engineer (L3) Data Substrate](https://www.wearedevelopers.com/jobs/ext/486993-software-engineer-l3-data-substrate) at **Twilio**
- [Software Engineer (L3) Data Substrate](https://www.wearedevelopers.com/jobs/ext/496550-software-engineer-l3-data-substrate) at **Twilio**
- [Software Engineer (L3) Data Substrate](https://www.wearedevelopers.com/jobs/ext/487210-software-engineer-l3-data-substrate) at **Twilio**
- [Software Engineer (L3) Data Substrate](https://www.wearedevelopers.com/jobs/ext/494111-software-engineer-l3-data-substrate) at **Twilio**