WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe • Jul 9, 2026 • Session details

Parquet, Delta, Iceberg & Ducklake - An introduction for developers

Matthias Niehoff

Monolithic databases and row-oriented formats are bottlenecking your analytical workloads. Master Parquet, Iceberg, and DuckDB to build performant, cost-effective reporting apps that decouple storage from compute.

Pause
Mute Enter Fullscreen
#1 about 1 min

Introduction to analytical data formats for software developers

Why software engineers need to understand the underlying infrastructure of data engineering pipelines.

#2 about 2 min

Separating storage and compute in data infrastructure

How decoupling storage blobs from query engines enables independent scaling and cost reduction.

#3 about 2 min

Limitations of row-oriented formats like CSV and JSON

Why parsing entire files row-by-row slows down data retrieval for specific analytical queries.

#4 about 2 min

Benefits and challenges of column-oriented storage

How strictly storing values by column accelerates targeted reads but complicates simple row appends.

#5 about 3 min

How Parquet metadata enables efficient data reading

How row groups, compression dictionaries, and footer statistics eliminate unnecessary file scanning through predicate pushdown.

#6 about 2 min

Shortcomings of standalone Parquet files for complex data

Why strict append-only Parquet struggles with ACID transactions, schema evolution, and localized data lookups.

#7 about 3 min

Introduction to open table formats built on Parquet

How tools like Iceberg, Delta, and Hudi add transaction management and mutability to base Parquet files.

#8 about 2 min

Tracking data modifications with Delta Lake transaction logs

How Delta Lake maintains a sequential log of commits and partial files to manage file mutations.

#9 about 2 min

Managing file structure using Apache Iceberg snapshots

How Iceberg relies on hierarchical manifest lists and metadata snapshot files rather than sequential logs.

#10 about 3 min

Abstracting file complexities using metadata catalogs

How catalog layers provide centralized access control, credential vending, and standardized table abstractions.

#11 about 2 min

Streamlining analytics infrastructure with Duck Lake and DuckDB

How DuckDB combines data computation and metadata database operations into a single embedded query engine.

#12 about 3 min

Choosing the right data format and catalog engine

When to use Parquet, Iceberg, Delta Lake, or DuckDB based on platform requirements and analytics complexity.

#13 about 2 min

Addressing slow write performance and point lookups

Why extensive metadata validation slows down writes and how experimental encoded formats attempt to solve this.

#14 about 2 min

Querying Parquet and Iceberg directly with DuckDB

How developers can use the DuckDB command-line interface or embedded library to cleanly query analytical files.

#15 about 2 min

Building cost-effective reporting apps without dedicated databases

How embedded compute directly analyzing persistent blob storage avoids continuous managed database costs.

#16 about 2 min

Summary of decoupling analytical compute and storage

Recap of how modern open table formats and querying layers enable highly customized, scalable analytics pipelines.

#17 about 4 min

Audience Q&A on data formats and engine tradeoffs

Audience discussions covering Avro usage scenarios, the rationale behind Parquet footers, and Elasticsearch efficiency.

Matching moments

1:16 min

Decoupling storage and compute with open lakehouse architectures

Max Fischer Max Fischer +1 · World Congress 2026 Europe

4:32 min

Evolution of centralized data architectures and open table formats

Matthias Niehoff Matthias Niehoff · World Congress 2024

4:30 min

Introducing data management and the shift to streaming

Mary Grygleski Mary Grygleski · LIVE

5:00 min

Q&A on analytical databases and market convergence

Andrey Abramov Andrey Abramov · World Congress 2026 Europe

3:52 min

Navigating data warehouse performance for application developers

Adnan Rahic · JS Congress

4:32 min

Evaluating open-source and cloud-based vector database vendors

Erik Bamberg · LIVE

More from World Congress 2026 Europe

Building the Nervous System of AI - Michael Kagan (NVIDIA) thumbnail
29:17
3 months ago

Building the Nervous System of AI - Michael Kagan (NVIDIA)

Michael Kagan, Sead Ahmetović

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote thumbnail
19:41
3 months ago

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote

Christian Heilmann

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend thumbnail
32:06
3 months ago

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

David vonThenen

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes thumbnail
35:13
3 months ago

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes

Irina Branovic

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬 thumbnail
24:23
3 months ago

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬

Jeremy Murray

Angstfreude - AI and Corporate Culture - The Thrill and the Threat thumbnail
30:26
3 months ago

Angstfreude - AI and Corporate Culture - The Thrill and the Threat

Axel Ebert, Karin Krobath

Related videos

Modern Data Architectures need Software Engineering thumbnail
29:25
World Congress 2024 • about 2 years ago

Modern Data Architectures need Software Engineering

Matthias Niehoff

How building an industry DBMS differs from building a research one thumbnail
30:34
World Congress 2023 • almost 3 years ago

How building an industry DBMS differs from building a research one

Markus Dreseler

Making Data Warehouses fast. A developer's story. thumbnail
42:16
JS Congress • almost 5 years ago

Making Data Warehouses fast. A developer's story.

Adnan Rahic

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone thumbnail
27:27
World Congress 2026 Europe • 3 months ago

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone

Daniël van Eeden, Mattias Jonsson

Swapping a Data Warehouse at Runtime: Zero-Downtime Migration Without Changing a Single Client thumbnail
30:28
World Congress 2026 Europe • 3 months ago

Swapping a Data Warehouse at Runtime: Zero-Downtime Migration Without Changing a Single Client

Max Fischer, Michael O'Toole

OLTP in the Lakehouse: Redefining Data for AI Workloads thumbnail
24:36
Europe 2026 Virtual • 3 months ago

OLTP in the Lakehouse: Redefining Data for AI Workloads

Oleksandra Bovkun

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.