World Congress 2025 • Aug 20, 2025 • Session details

Let's Get Aggregated: Custom UDAFs in Spark

Nikolai Nikolaev

Stop letting expensive data shuffles bottleneck your Spark workflows. Custom UDAFs pre-aggregate data locally, condensing complex analytics into a single highly performant shuffle.

Let's Get Aggregated: Custom UDAFs in Spark thumbnail

Checking access…

Playback and chapters load privately for Free videos.

Matching moments

2:29 min

Optimizing Spark performance for large data volumes

Alan Mazankiewicz · LIVE

3:33 min

Refactoring data science workflows using Rapids QDF and Pandas

Paul Graham

Paul Graham · LIVE

3:49 min

Understanding RDDs, DataFrames, and Datasets in Spark

Ayon Roy · LIVE

2:04 min

Comparing offline data analytics with online stream processing

Artem Volk

Artem Volk +1 · World Congress 2024

4:06 min

Pre-calculating data aggregates using event-driven methodologies

Luka Samaržija +1 · LIVE

2:34 min

Capabilities of the Apache Spark processing engine

Ayon Roy · LIVE

More from World Congress 2025

GenAI Security: Navigating the Unseen Iceberg thumbnail

GenAI Security: Navigating the Unseen Iceberg

Maish Saidel-Keesing · AWS

Behind the Code: How Women Are Powering the Future of AI thumbnail

Behind the Code: How Women Are Powering the Future of AI

Alexandra Wudel +2

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise? thumbnail

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise?

Justus Lumpe +3

AI in Leadership: How Technology is Reshaping Executive Roles thumbnail

AI in Leadership: How Technology is Reshaping Executive Roles

Bernd Datler +3

The AI-Ready Stack: Rethinking the Engineering Org of the Future thumbnail

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Alex Laubscher +3

AI Sovereignty: What Does It Take? thumbnail

AI Sovereignty: What Does It Take?

Boris Hecker +3

Upcoming sessions on this topic

React Under Fire: Rendering Performance for Streaming Agent UIs

Abhirupa Mitra

Slack

Senior Member of Technical Staff

The 200,000 line code repository: My AI wrote 1000 lines. I needed 100.

Hardik Gupta

IBM

Software Developer

Talk to Your Data Warehouse: Building Real-Time Analyst Agents with ADK and BigQuery MCP

Priyanka Malhotra

Google Cloud

Platform Customer Engineer

Data Quality Is the Hidden System That Keeps Modern Data Platforms Alive

Rakshika Prasad

Schneider Electric

Data Engineer

Build Enterprise-Grade Agentic Workflows with AI Agents and Distributed Transactions

Bharath MC

Oracle

Principal Platform Software Engineer
Pruthvithej R

Oracle

Lead Principal Platform Software Engineer

How We Cut Our API's p99 Latency from Minutes to Under a Second

Deepak Agrawal

Atlassian

Principal Software Engineer

Related videos

PySpark - Combining Machine Learning & Big Data thumbnail

PySpark - Combining Machine Learning & Big Data

Ayon Roy

LIVE · Oct 2020

Make Your Data FABulous thumbnail

Make Your Data FABulous

Philipp Krenn

World Congress · 2022

Making Data Warehouses fast. A developer's story. thumbnail

Making Data Warehouses fast. A developer's story.

Adnan Rahic

JS Congress · 2021

From Model to Metal: An Open Source Stack for Accelerating Intelligence thumbnail

From Model to Metal: An Open Source Stack for Accelerating Intelligence

Andrew Wafaa · Arm

World Congress · 2025

Parquet, Delta, Iceberg & Ducklake - An introduction for developers thumbnail

Parquet, Delta, Iceberg & Ducklake - An introduction for developers

Matthias Niehoff · codecentric

World Congress Europe · 2026

Real-Time Data Platforms at Trillion-Event Scale thumbnail

Real-Time Data Platforms at Trillion-Event Scale

Diptamay Sanyal

World Congress North America · 2026

Related articles

BB

Benedikt Bischof

Making Data Warehouses Fast: A Developer’s Story

Welcome to this issue of the WeAreDevelopers Live Talk series. This article recaps an interesting talk by Adnan Rahic who teaches the audience how to make data warehouses.About the Speaker: Adnan i...

over 4 years ago

Making Data Warehouses Fast: A Developer’s Story

DC

Daniel Cranney

Infer Once, Run Everywhere: Zero-Token Architecture

“You are all having way too much fun burning way too many tokens”. Kelsey Hightower kicked off his talk at the WeAreDevelopers World Congress North America by calling out the developers in attend...

27 days ago

A close-up on a screen showing a terminal with 'today's token spend: 0' on it.

AA

Ali Alp

Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)

Why Event-Driven Architecture Isn’t About Speed—And When You Actually Need It We’ve all seen it happen. An engineering team wants to feel like they belong in the “cool kids” club, so they grab Apac...

3 months ago

A developer workstation showing a code editor on one monitor, and a diagram on another

DS

Dhannush Subramani

Top Big Data Technologies That You Need to Know

Quite possibly the most advancing technology in today’s world is Big Data. It is a term that is utilised to portray an assortment of information that is immense in size and is dramatically expandin...

over 4 years ago

Top Big Data Technologies That You Need to Know

From learning to earning

Jobs that call for the skills explored in this talk.

Senior Data & AI Architect

reeeliance IM GmbH

Berlin, Germany

Expert Hybrid · 3 days on-site · Remote Python Microsoft SQL Server Cloud (AWS/Google/Azure)

Data Analyst

Experienced $227k Remote Github Airflow Power BI

Senior Analytics Engineer

San Francisco, CA, United States

Expert $149k–187k Remote Presto Twilio DataOps

Analytics Engineer

Port Angeles, WA, United States

Expert $149k–187k Remote Presto Twilio DataOps

Senior Software Engineer - Semantic Data Lake (Open)SDE3

Bengaluru, India

Expert Hybrid · 4 days on-site Scala Python Terraform

Deal Data Technology & Analytics, Senior Associate

San Jose, CA, United States

Expert $77k–202k Spacy Claude OpenAI