WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2025 • Aug 20, 2025 • Session details

Let's Get Aggregated: Custom UDAFs in Spark

Nikolai Nikolaev

Stop letting expensive data shuffles bottleneck your Spark workflows. Custom UDAFs pre-aggregate data locally, condensing complex analytics into a single highly performant shuffle.

Pause
Mute Enter Fullscreen
#1 about 3 min

Motivations for custom aggregation functions in large datasets

Standard aggregation functions often fall short when processing complex analytical workloads across billion-row datasets.

#2 about 2 min

Defining the core methods of user-defined aggregator functions

Custom aggregations require implementing zero, reduce, merge, and finish functions alongside appropriate encoders.

#3 about 2 min

Understanding distributed execution and the necessity of pre-aggregation

Executing custom aggregations in a distributed framework minimizes network data shuffling by pre-aggregating data locally on executors.

#4 about 3 min

Analyzing baseline performance issues in standard word count techniques

Standard approaches to grouping and windowing functions generate multiple expensive data shuffles within the execution plan.

#5 about 4 min

Designing custom aggregation workflows for efficient frequency mapping

Defining specialized types and mapping logic allows for streamlined reduction and merging phases without repetitive native queries.

#6 about 3 min

Implementing core zero, reduce, merge, and finish functions

Developing each stage involves tracking frequencies in mapped structures and handling null checks within encoders.

#7 about 2 min

Comparing execution plans to benchmark reduced data shuffling

Executing the user-defined aggregation produces an optimized execution plan featuring only a single data shuffle operation.

#8 about 2 min

Leveraging complex data structures in production aggregation workflows

Utilizing case classes as buffers and output values enables calculating multiple metrics reliably within a single operation.

#9 about 3 min

Optimal scenarios for deploying custom defined aggregator functions

Custom functions optimize complex aggregation performance, enable code reusability across teams, and integrate seamlessly with Spark SQL APIs.

Matching moments

3:33 min

Refactoring data science workflows using Rapids QDF and Pandas

Paul Graham Paul Graham · LIVE

3:49 min

Understanding RDDs, DataFrames, and Datasets in Spark

Ayon Roy · LIVE

2:04 min

Comparing offline data analytics with online stream processing

Artem Volk Artem Volk +1 · World Congress 2024

4:06 min

Pre-calculating data aggregates using event-driven methodologies

Luka Samaržija +1 · LIVE

2:34 min

Capabilities of the Apache Spark processing engine

Ayon Roy · LIVE

1:21 min

Summary of decoupling analytical compute and storage

Matthias Niehoff Matthias Niehoff · World Congress 2026 Europe

More from World Congress 2025

GenAI Security: Navigating the Unseen Iceberg thumbnail
24:33
11 months ago

GenAI Security: Navigating the Unseen Iceberg

Maish Saidel-Keesing

Behind the Code: How Women Are Powering the Future of AI thumbnail
30:28
about 1 year ago

Behind the Code: How Women Are Powering the Future of AI

Alexandra Wudel, Laura Moritz, Madalina Florean

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise? thumbnail
28:12
about 1 year ago

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise?

Justus Lumpe, Konstantin Vinogradov, Nils Eiteneyer, Stephan Wirris

AI in Leadership: How Technology is Reshaping Executive Roles thumbnail
28:19
about 1 year ago

AI in Leadership: How Technology is Reshaping Executive Roles

Bernd Datler, Jasmin Kaiser, Jeff Hausmann, Sonja Alvarez

The AI-Ready Stack: Rethinking the Engineering Org of the Future thumbnail
28:09
about 1 year ago

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Alex Laubscher, Jan Oberhauser, Keno Dreßel, Mirko Novakovic

AI Sovereignty: What Does It Take? thumbnail
29:13
about 1 year ago

AI Sovereignty: What Does It Take?

Boris Hecker, Clemens Wasner, Robin Hermann, Sonja Alvarez

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 24, 2026 · 11:10–11:15

Outdoor Stage

Architecting the 100X SDLC: Building Production Trust into AI-Assisted Delivery

Ranjan Parthasarathy

Founder, CPTO/CEO at AXIOMSTUDIO.AI

Ranjan Parthasarathy
Open session

World Congress 2026 North America

September 25, 2026 · 13:30–14:00

Stage 4

From Model Selection to Smart Routing: How to Use the Right LLM for Every Task

Viktoria Semaan

Principal Technical Evangelist at Databricks

Viktoria Semaan
Open session

World Congress 2026 North America

September 24, 2026 · 11:00–11:30

Stage 1

Application-Defined Compute: Rethinking Infrastructure for AI Applications

Anurag Goel

Founder and CEO of Render

Anurag Goel
Open session

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 1

Anatomy of an AI Request: Where Latency and Cost Are Really Born

Dan Fu

VP of Kernels at Together AI

Dan Fu
Open session

World Congress 2026 North America

September 25, 2026 · 09:00–09:30

Stage 7

AI Agents are Only as Smart as their Context: Building a Real-Time Context Engine at Intuit

Bharat Patel

Lead Software Engineer at Intuit

Bharat Patel
Open session

World Congress 2026 North America

September 24, 2026 · 17:30–18:00

Stage 3

Scaling Distributed Queues for AI workloads

Jasmit Kaur Saluja

Software Engineer at Meta Platforms Inc

Jasmit Kaur Saluja

Related videos

PySpark - Combining Machine Learning & Big Data thumbnail
43:26
LIVE • almost 6 years ago

PySpark - Combining Machine Learning & Big Data

Ayon Roy

Make Your Data FABulous thumbnail
28:39
World Congress 2022 • about 4 years ago

Make Your Data FABulous

Philipp Krenn

Making Data Warehouses fast. A developer's story. thumbnail
42:16
JS Congress • almost 5 years ago

Making Data Warehouses fast. A developer's story.

Adnan Rahic

From Model to Metal: An Open Source Stack for Accelerating Intelligence thumbnail
17:18
World Congress 2025 • about 1 year ago

From Model to Metal: An Open Source Stack for Accelerating Intelligence

Andrew Wafaa

Parquet, Delta, Iceberg & Ducklake - An introduction for developers thumbnail
28:42
World Congress 2026 Europe • about 2 months ago

Parquet, Delta, Iceberg & Ducklake - An introduction for developers

Matthias Niehoff

Leveraging Real time data in FSIs thumbnail
41:51
LIVE • almost 3 years ago

Leveraging Real time data in FSIs

Tim Faulkes

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.