WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe • Jul 9, 2026 • Session details

Parquet, Delta, Iceberg & Ducklake - An introduction for developers

Matthias Niehoff

Monolithic databases and row-oriented formats are bottlenecking your analytical workloads. Master Parquet, Iceberg, and DuckDB to build performant, cost-effective reporting apps that decouple storage from compute.

Pause
Mute Enter Fullscreen
#1 about 1 min

Introduction to analytical data formats for software developers

Why software engineers need to understand the underlying infrastructure of data engineering pipelines.

#2 about 2 min

Separating storage and compute in data infrastructure

How decoupling storage blobs from query engines enables independent scaling and cost reduction.

#3 about 2 min

Limitations of row-oriented formats like CSV and JSON

Why parsing entire files row-by-row slows down data retrieval for specific analytical queries.

#4 about 2 min

Benefits and challenges of column-oriented storage

How strictly storing values by column accelerates targeted reads but complicates simple row appends.

#5 about 3 min

How Parquet metadata enables efficient data reading

How row groups, compression dictionaries, and footer statistics eliminate unnecessary file scanning through predicate pushdown.

#6 about 2 min

Shortcomings of standalone Parquet files for complex data

Why strict append-only Parquet struggles with ACID transactions, schema evolution, and localized data lookups.

#7 about 3 min

Introduction to open table formats built on Parquet

How tools like Iceberg, Delta, and Hudi add transaction management and mutability to base Parquet files.

#8 about 2 min

Tracking data modifications with Delta Lake transaction logs

How Delta Lake maintains a sequential log of commits and partial files to manage file mutations.

#9 about 2 min

Managing file structure using Apache Iceberg snapshots

How Iceberg relies on hierarchical manifest lists and metadata snapshot files rather than sequential logs.

#10 about 3 min

Abstracting file complexities using metadata catalogs

How catalog layers provide centralized access control, credential vending, and standardized table abstractions.

#11 about 2 min

Streamlining analytics infrastructure with Duck Lake and DuckDB

How DuckDB combines data computation and metadata database operations into a single embedded query engine.

#12 about 3 min

Choosing the right data format and catalog engine

When to use Parquet, Iceberg, Delta Lake, or DuckDB based on platform requirements and analytics complexity.

#13 about 2 min

Addressing slow write performance and point lookups

Why extensive metadata validation slows down writes and how experimental encoded formats attempt to solve this.

#14 about 2 min

Querying Parquet and Iceberg directly with DuckDB

How developers can use the DuckDB command-line interface or embedded library to cleanly query analytical files.

#15 about 2 min

Building cost-effective reporting apps without dedicated databases

How embedded compute directly analyzing persistent blob storage avoids continuous managed database costs.

#16 about 2 min

Summary of decoupling analytical compute and storage

Recap of how modern open table formats and querying layers enable highly customized, scalable analytics pipelines.

#17 about 4 min

Audience Q&A on data formats and engine tradeoffs

Audience discussions covering Avro usage scenarios, the rationale behind Parquet footers, and Elasticsearch efficiency.

Matching moments

1:16 min

Decoupling storage and compute with open lakehouse architectures

Max Fischer Max Fischer +1 · WWC Europe 2026

4:32 min

Evolution of centralized data architectures and open table formats

Matthias Niehoff Matthias Niehoff · WWC 2024

4:30 min

Introducing data management and the shift to streaming

Mary Grygleski Mary Grygleski · LIVE

5:00 min

Q&A on analytical databases and market convergence

Andrey Abramov Andrey Abramov · WWC Europe 2026

3:52 min

Navigating data warehouse performance for application developers

Adnan Rahic · JS Congress

4:32 min

Evaluating open-source and cloud-based vector database vendors

Erik Bamberg · LIVE

More from World Congress 2026 Europe

Building the Nervous System of AI - Michael Kagan (NVIDIA) thumbnail
29:17
about 1 month ago

Building the Nervous System of AI - Michael Kagan (NVIDIA)

Michael Kagan, Sead Ahmetović

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote thumbnail
19:41
about 1 month ago

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote

Christian Heilmann

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend thumbnail
32:06
about 1 month ago

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

David vonThenen

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes thumbnail
35:13
about 1 month ago

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes

Irina Branovic

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬 thumbnail
24:23
about 1 month ago

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬

Jeremy Murray

Angstfreude - AI and Corporate Culture - The Thrill and the Threat thumbnail
30:26
about 1 month ago

Angstfreude - AI and Corporate Culture - The Thrill and the Threat

Axel Ebert, Karin Krobath

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Databases in the Agent Era

Monica Sarbu

Founder and CEO of xata.io

Monica Sarbu
Open session

World Congress 2026 North America

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

Beyond SQL Generation: How to Teach Agents What Your Database Actually Means

Kris Jenkins

Lead Developer Advocate at Snowflake

Kris Jenkins
Open session

World Congress 2026 North America

AI Agents are Only as Smart as their Context: Building a Real-Time Context Engine at Intuit

Bharat Patel

Lead Software Engineer at Intuit

Bharat Patel
Open session

World Congress 2026 North America

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan
Open session

World Congress 2026 North America

public void saveMoney(AI): The Developer's Guide to Unit Economics

Hrushikesh Pokala

Senior Software Engineer Lead at Equifax

Hrushikesh Pokala

Related videos

Modern Data Architectures need Software Engineering thumbnail
29:25
WWC 2024 • almost 2 years ago

Modern Data Architectures need Software Engineering

Matthias Niehoff

How building an industry DBMS differs from building a research one thumbnail
30:34
WWC 2023 • almost 3 years ago

How building an industry DBMS differs from building a research one

Markus Dreseler

Making Data Warehouses fast. A developer's story. thumbnail
42:16
JS Congress • over 4 years ago

Making Data Warehouses fast. A developer's story.

Adnan Rahic

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone thumbnail
27:27
WWC Europe 2026 • about 1 month ago

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone

Daniël van Eeden, Mattias Jonsson

Swapping a Data Warehouse at Runtime: Zero-Downtime Migration Without Changing a Single Client thumbnail
30:28
WWC Europe 2026 • about 1 month ago

Swapping a Data Warehouse at Runtime: Zero-Downtime Migration Without Changing a Single Client

Max Fischer, Michael O'Toole

From Messy Queries to Scalable Systems - How Data Engineering actually works thumbnail
31:15
WWC Europe 2026 • about 1 month ago

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.