WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe • Jul 9, 2026 • Session details

Parquet, Delta, Iceberg & Ducklake - An introduction for developers

Matthias Niehoff

Monolithic databases and row-oriented formats are bottlenecking your analytical workloads. Master Parquet, Iceberg, and DuckDB to build performant, cost-effective reporting apps that decouple storage from compute.

Pause
Mute Enter Fullscreen
#1 about 1 min

Introduction to analytical data formats for software developers

Why software engineers need to understand the underlying infrastructure of data engineering pipelines.

#2 about 2 min

Separating storage and compute in data infrastructure

How decoupling storage blobs from query engines enables independent scaling and cost reduction.

#3 about 2 min

Limitations of row-oriented formats like CSV and JSON

Why parsing entire files row-by-row slows down data retrieval for specific analytical queries.

#4 about 2 min

Benefits and challenges of column-oriented storage

How strictly storing values by column accelerates targeted reads but complicates simple row appends.

#5 about 3 min

How Parquet metadata enables efficient data reading

How row groups, compression dictionaries, and footer statistics eliminate unnecessary file scanning through predicate pushdown.

#6 about 2 min

Shortcomings of standalone Parquet files for complex data

Why strict append-only Parquet struggles with ACID transactions, schema evolution, and localized data lookups.

#7 about 3 min

Introduction to open table formats built on Parquet

How tools like Iceberg, Delta, and Hudi add transaction management and mutability to base Parquet files.

#8 about 2 min

Tracking data modifications with Delta Lake transaction logs

How Delta Lake maintains a sequential log of commits and partial files to manage file mutations.

#9 about 2 min

Managing file structure using Apache Iceberg snapshots

How Iceberg relies on hierarchical manifest lists and metadata snapshot files rather than sequential logs.

#10 about 3 min

Abstracting file complexities using metadata catalogs

How catalog layers provide centralized access control, credential vending, and standardized table abstractions.

#11 about 2 min

Streamlining analytics infrastructure with Duck Lake and DuckDB

How DuckDB combines data computation and metadata database operations into a single embedded query engine.

#12 about 3 min

Choosing the right data format and catalog engine

When to use Parquet, Iceberg, Delta Lake, or DuckDB based on platform requirements and analytics complexity.

#13 about 2 min

Addressing slow write performance and point lookups

Why extensive metadata validation slows down writes and how experimental encoded formats attempt to solve this.

#14 about 2 min

Querying Parquet and Iceberg directly with DuckDB

How developers can use the DuckDB command-line interface or embedded library to cleanly query analytical files.

#15 about 2 min

Building cost-effective reporting apps without dedicated databases

How embedded compute directly analyzing persistent blob storage avoids continuous managed database costs.

#16 about 2 min

Summary of decoupling analytical compute and storage

Recap of how modern open table formats and querying layers enable highly customized, scalable analytics pipelines.

#17 about 4 min

Audience Q&A on data formats and engine tradeoffs

Audience discussions covering Avro usage scenarios, the rationale behind Parquet footers, and Elasticsearch efficiency.

Matching moments

1:16 min

Decoupling storage and compute with open lakehouse architectures

Max Fischer Max Fischer +1 · World Congress 2026 Europe

4:32 min

Evolution of centralized data architectures and open table formats

Matthias Niehoff Matthias Niehoff · World Congress 2024

4:30 min

Introducing data management and the shift to streaming

Mary Grygleski Mary Grygleski · LIVE

5:00 min

Q&A on analytical databases and market convergence

Andrey Abramov Andrey Abramov · World Congress 2026 Europe

3:52 min

Navigating data warehouse performance for application developers

Adnan Rahic · JS Congress

4:32 min

Evaluating open-source and cloud-based vector database vendors

Erik Bamberg · LIVE

More from World Congress 2026 Europe

Building the Nervous System of AI - Michael Kagan (NVIDIA) thumbnail
29:17
about 2 months ago

Building the Nervous System of AI - Michael Kagan (NVIDIA)

Michael Kagan, Sead Ahmetović

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote thumbnail
19:41
about 2 months ago

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote

Christian Heilmann

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend thumbnail
32:06
about 2 months ago

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

David vonThenen

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes thumbnail
35:13
about 2 months ago

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes

Irina Branovic

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬 thumbnail
24:23
about 2 months ago

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬

Jeremy Murray

Angstfreude - AI and Corporate Culture - The Thrill and the Threat thumbnail
30:26
about 2 months ago

Angstfreude - AI and Corporate Culture - The Thrill and the Threat

Axel Ebert, Karin Krobath

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 24, 2026 · 11:40–12:10

Stage 7

Clean Rooms Demystified: Architecture and Patterns for Privacy-Safe Data Collaboration

Anurag Malik

Data Tech Lead @ Intuit

Anurag Malik
Open session

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 3

Real-Time Data Platforms at Trillion-Event Scale

Diptamay Sanyal

Principal Engineer | Data, AI & Cybersecurity Platforms

Diptamay Sanyal
Open session

World Congress 2026 North America

September 25, 2026 · 16:10–16:40

Stage 1

The Five Percent Club: The Culture and Technological Shift Behind Successful AI Deployments

Tara Hernandez

Tara Hernandez, VP of Developer Productivity at MongoDB

Tara Hernandez
Open session

World Congress 2026 North America

September 24, 2026 · 15:30–16:00

Tech Leaders Stage

Transforming Enterprise AI with Tabular Foundation Models

Leo Grinsztajn, Tom Palczewski, Yaad Oren

Leo Grinsztajn
Tom Palczewski
Yaad Oren
Open session

World Congress 2026 North America

September 24, 2026 · 14:50–15:20

Stage 2

Databases in the Agent Era

Monica Sarbu

Founder and CEO of xata.io

Monica Sarbu
Open session

World Congress 2026 North America

September 24, 2026 · 16:50–17:20

Stage 9

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs

Related videos

Modern Data Architectures need Software Engineering thumbnail
29:25
World Congress 2024 • about 2 years ago

Modern Data Architectures need Software Engineering

Matthias Niehoff

How building an industry DBMS differs from building a research one thumbnail
30:34
World Congress 2023 • almost 3 years ago

How building an industry DBMS differs from building a research one

Markus Dreseler

Making Data Warehouses fast. A developer's story. thumbnail
42:16
JS Congress • almost 5 years ago

Making Data Warehouses fast. A developer's story.

Adnan Rahic

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone thumbnail
27:27
World Congress 2026 Europe • about 2 months ago

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone

Daniël van Eeden, Mattias Jonsson

Swapping a Data Warehouse at Runtime: Zero-Downtime Migration Without Changing a Single Client thumbnail
30:28
World Congress 2026 Europe • about 2 months ago

Swapping a Data Warehouse at Runtime: Zero-Downtime Migration Without Changing a Single Client

Max Fischer, Michael O'Toole

OLTP in the Lakehouse: Redefining Data for AI Workloads thumbnail
24:36
Europe 2026 Virtual • 2 months ago

OLTP in the Lakehouse: Redefining Data for AI Workloads

Oleksandra Bovkun

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.