WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2025 • Aug 20, 2025 • Session details

Let's Get Aggregated: Custom UDAFs in Spark

Nikolai Nikolaev

Stop letting expensive data shuffles bottleneck your Spark workflows. Custom UDAFs pre-aggregate data locally, condensing complex analytics into a single highly performant shuffle.

Pause
Mute Enter Fullscreen
#1 about 3 min

Motivations for custom aggregation functions in large datasets

Standard aggregation functions often fall short when processing complex analytical workloads across billion-row datasets.

#2 about 2 min

Defining the core methods of user-defined aggregator functions

Custom aggregations require implementing zero, reduce, merge, and finish functions alongside appropriate encoders.

#3 about 2 min

Understanding distributed execution and the necessity of pre-aggregation

Executing custom aggregations in a distributed framework minimizes network data shuffling by pre-aggregating data locally on executors.

#4 about 3 min

Analyzing baseline performance issues in standard word count techniques

Standard approaches to grouping and windowing functions generate multiple expensive data shuffles within the execution plan.

#5 about 4 min

Designing custom aggregation workflows for efficient frequency mapping

Defining specialized types and mapping logic allows for streamlined reduction and merging phases without repetitive native queries.

#6 about 3 min

Implementing core zero, reduce, merge, and finish functions

Developing each stage involves tracking frequencies in mapped structures and handling null checks within encoders.

#7 about 2 min

Comparing execution plans to benchmark reduced data shuffling

Executing the user-defined aggregation produces an optimized execution plan featuring only a single data shuffle operation.

#8 about 2 min

Leveraging complex data structures in production aggregation workflows

Utilizing case classes as buffers and output values enables calculating multiple metrics reliably within a single operation.

#9 about 3 min

Optimal scenarios for deploying custom defined aggregator functions

Custom functions optimize complex aggregation performance, enable code reusability across teams, and integrate seamlessly with Spark SQL APIs.

Matching moments

3:33 min

Refactoring data science workflows using Rapids QDF and Pandas

Paul Graham Paul Graham · LIVE

3:49 min

Understanding RDDs, DataFrames, and Datasets in Spark

Ayon Roy · LIVE

2:04 min

Comparing offline data analytics with online stream processing

Artem Volk Artem Volk +1 · WWC 2024

4:06 min

Pre-calculating data aggregates using event-driven methodologies

Luka Samaržija +1 · LIVE

2:34 min

Capabilities of the Apache Spark processing engine

Ayon Roy · LIVE

1:21 min

Summary of decoupling analytical compute and storage

Matthias Niehoff Matthias Niehoff · WWC Europe 2026

More from World Congress 2025

GenAI Security: Navigating the Unseen Iceberg thumbnail
24:33
10 months ago

GenAI Security: Navigating the Unseen Iceberg

Maish Saidel-Keesing

Behind the Code: How Women Are Powering the Future of AI thumbnail
30:28
12 months ago

Behind the Code: How Women Are Powering the Future of AI

Alexandra Wudel, Laura Moritz, Madalina Florean

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise? thumbnail
28:12
12 months ago

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise?

Justus Lumpe, Konstantin Vinogradov, Nils Eiteneyer, Stephan Wirris

AI in Leadership: How Technology is Reshaping Executive Roles thumbnail
28:19
12 months ago

AI in Leadership: How Technology is Reshaping Executive Roles

Bernd Datler, Jasmin Kaiser, Jeff Hausmann, Sonja Alvarez

The AI-Ready Stack: Rethinking the Engineering Org of the Future thumbnail
28:09
12 months ago

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Alex Laubscher, Jan Oberhauser, Keno Dreßel, Mirko Novakovic

AI Sovereignty: What Does It Take? thumbnail
29:13
12 months ago

AI Sovereignty: What Does It Take?

Boris Hecker, Clemens Wasner, Robin Hermann, Sonja Alvarez

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Ship 10x Faster: AI-Powered Development with Claude Code and MCP Tools

Viktoria Semaan

Principal Technical Evangelist at Databricks

Viktoria Semaan
Open session

World Congress 2026 North America

AI Agents are Only as Smart as their Context: Building a Real-Time Context Engine at Intuit

Bharat Patel

Lead Software Engineer at Intuit

Bharat Patel
Open session

World Congress 2026 North America

The spectrum of agentic coding: From vibe coding to high-quality software engineering

YK Sugi

Developer Experience Manager at Eventual

YK Sugi
Open session

World Congress 2026 North America

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

public void saveMoney(AI): The Developer's Guide to Unit Economics

Hrushikesh Pokala

Senior Software Engineer Lead at Equifax

Hrushikesh Pokala
Open session

World Congress 2026 North America

Compute for your AI model: GPUs, LPUs, TPUs and beyond..

Kushaagra Goyal

Tech Lead at Rubrik, ex-CTO at Gan.AI, ex-Databricks

Kushaagra Goyal

Related videos

PySpark - Combining Machine Learning & Big Data thumbnail
43:26
LIVE • almost 6 years ago

PySpark - Combining Machine Learning & Big Data

Ayon Roy

Make Your Data FABulous thumbnail
28:39
WWC 2022 • about 4 years ago

Make Your Data FABulous

Philipp Krenn

Making Data Warehouses fast. A developer's story. thumbnail
42:16
JS Congress • over 4 years ago

Making Data Warehouses fast. A developer's story.

Adnan Rahic

From Model to Metal: An Open Source Stack for Accelerating Intelligence thumbnail
17:18
WWC 2025 • 12 months ago

From Model to Metal: An Open Source Stack for Accelerating Intelligence

Andrew Wafaa

Parquet, Delta, Iceberg & Ducklake - An introduction for developers thumbnail
28:42
WWC Europe 2026 • about 1 month ago

Parquet, Delta, Iceberg & Ducklake - An introduction for developers

Matthias Niehoff

Leveraging Real time data in FSIs thumbnail
41:51
LIVE • over 2 years ago

Leveraging Real time data in FSIs

Tim Faulkes

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.