> Markdown version of [/videos/100075-parquet-delta-iceberg-ducklake-an-introduction-for-developers?t=1458](https://www.wearedevelopers.com/videos/100075-parquet-delta-iceberg-ducklake-an-introduction-for-developers?t=1458). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Parquet, Delta, Iceberg & Ducklake - An introduction for developers

Monolithic databases and row-oriented formats are bottlenecking your analytical workloads. Master Parquet, Iceberg, and DuckDB to build performant, cost-effective reporting apps that decouple storage from compute.

- **Speakers:** [Matthias Niehoff](https://www.wearedevelopers.com/@matthias-niehoff)
- **Event:** World Congress 2026 Europe
- **Published:** July 9, 2026
- **Duration:** 28:42
- **URL:** https://www.wearedevelopers.com/videos/100075-parquet-delta-iceberg-ducklake-an-introduction-for-developers

## Summary

Data engineering has moved away from monolithic databases toward decoupled storage and compute, rendering traditional row-oriented formats like CSV and JSON highly inefficient for analytical workloads. As software engineers increasingly build data-intensive applications, understanding modern storage architecture is critical. Scanning entire rows to aggregate a single column wastes both time and compute resources. The solution lies in column-oriented storage formats like Apache Parquet, which groups data by columns and embeds rich metadata—such as dictionary encoding and min/max statistics—within footers. This structure enables "predicate pushdown," allowing systems to skip irrelevant blocks of data entirely and read only what is strictly necessary.

While highly optimized for reading, Parquet remains fundamentally append-only, lacking schema evolution, ACID compliance, and safe concurrent-write capabilities. These limitations birthed Open Table Formats (OTFs) like Apache Iceberg and Delta Lake. These formats layer snapshot metadata, transactional logs, and data catalogs (such as Apache Polaris or Unity Catalog) on top of raw Parquet files to provide database-like reliability over cloud blob storage. By upgrading from raw files to tables, developers enable multi-engine interoperability, where tools ranging from Apache Spark to Trino can reliably read the same underlying data without corrupting the state.

Choosing the right abstraction layer depends heavily on the project's scale and ecosystem. Iceberg dominates widespread analytical adoption, while Delta shines in Databricks environments. However, for many developer-led projects, navigating complex metadata catalogs is over-engineered. Tools like DuckDB offer a technically superior, lightweight compromise. By embedding DuckDB directly into an application, engineers can query Parquet paths on cloud storage natively via SQL without provisioning a persistent database server. Ultimately, developers can leverage these open standards to build performant, cost-effective reporting applications that keep transactional databases safe from heavy analytical burdens.

**Keywords:** column-oriented storage, apache parquet optimization, apache iceberg metadata, delta lake transaction logs, duckdb embedded analytics, decoupled storage and compute, predicate pushdown, open table formats, ACID compliance, data catalog layer, schema evolution, dictionary encoding, analytical data processing, blob storage querying

## Chapters

1. **Introduction to analytical data formats for software developers** (00:00) — Why software engineers need to understand the underlying infrastructure of data engineering pipelines.
1. **Separating storage and compute in data infrastructure** (00:56) — How decoupling storage blobs from query engines enables independent scaling and cost reduction.
1. **Limitations of row-oriented formats like CSV and JSON** (02:19) — Why parsing entire files row-by-row slows down data retrieval for specific analytical queries.
1. **Benefits and challenges of column-oriented storage** (03:39) — How strictly storing values by column accelerates targeted reads but complicates simple row appends.
1. **How Parquet metadata enables efficient data reading** (04:55) — How row groups, compression dictionaries, and footer statistics eliminate unnecessary file scanning through predicate pushdown.
1. **Shortcomings of standalone Parquet files for complex data** (07:45) — Why strict append-only Parquet struggles with ACID transactions, schema evolution, and localized data lookups.
1. **Introduction to open table formats built on Parquet** (08:47) — How tools like Iceberg, Delta, and Hudi add transaction management and mutability to base Parquet files.
1. **Tracking data modifications with Delta Lake transaction logs** (10:51) — How Delta Lake maintains a sequential log of commits and partial files to manage file mutations.
1. **Managing file structure using Apache Iceberg snapshots** (12:31) — How Iceberg relies on hierarchical manifest lists and metadata snapshot files rather than sequential logs.
1. **Abstracting file complexities using metadata catalogs** (13:43) — How catalog layers provide centralized access control, credential vending, and standardized table abstractions.
1. **Streamlining analytics infrastructure with Duck Lake and DuckDB** (16:31) — How DuckDB combines data computation and metadata database operations into a single embedded query engine.
1. **Choosing the right data format and catalog engine** (18:16) — When to use Parquet, Iceberg, Delta Lake, or DuckDB based on platform requirements and analytics complexity.
1. **Addressing slow write performance and point lookups** (20:18) — Why extensive metadata validation slows down writes and how experimental encoded formats attempt to solve this.
1. **Querying Parquet and Iceberg directly with DuckDB** (21:28) — How developers can use the DuckDB command-line interface or embedded library to cleanly query analytical files.
1. **Building cost-effective reporting apps without dedicated databases** (23:07) — How embedded compute directly analyzing persistent blob storage avoids continuous managed database costs.
1. **Summary of decoupling analytical compute and storage** (24:18) — Recap of how modern open table formats and querying layers enable highly customized, scalable analytics pipelines.
1. **Audience Q&A on data formats and engine tradeoffs** (25:39) — Audience discussions covering Avro usage scenarios, the rationale behind Parquet footers, and Elasticsearch efficiency.

## Related Moments

- [Decoupling storage and compute with open lakehouse architectures](https://www.wearedevelopers.com/videos/100311-swapping-a-data-warehouse-at-runtime-zero-downtime-migration-without-changing-a-single-client) (from "Swapping a Data Warehouse at Runtime: Zero-Downtime Migration Without Changing a Single Client")
- [Evolution of centralized data architectures and open table formats](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) (from "Modern Data Architectures need Software Engineering")
- [Introducing data management and the shift to streaming](https://www.wearedevelopers.com/videos/538-event-messaging-and-streaming-with-apache-pulsar) (from "Event Messaging and Streaming with Apache Pulsar")
- [Q&A on analytical databases and market convergence](https://www.wearedevelopers.com/videos/100212-olap-for-ai-applications-and-why-you-should-care) (from "OLAP for AI Applications and why you should care")
- [Navigating data warehouse performance for application developers](https://www.wearedevelopers.com/videos/302-making-data-warehouses-fast-a-developer-s-story) (from "Making Data Warehouses fast. A developer's story.")
- [Evaluating open-source and cloud-based vector database vendors](https://www.wearedevelopers.com/videos/830-what-comes-after-chatgpt-vector-databases-the-simple-and-powerful-future-of-ml) (from "What comes after ChatGPT? Vector Databases - the Simple and powerful future of ML?")

## Related Articles

- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [What does the history of data storage tell us about the future?](https://www.wearedevelopers.com/magazine/495-what-does-the-history-of-data-storage-tell-us-about-the-future)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Dev Digest 139 - Soft and hard queries](https://www.wearedevelopers.com/magazine/487-dev-digest-139-soft-and-hard-queries)

## Related Jobs

- [Senior Data Engineer](https://www.wearedevelopers.com/jobs/ext/1589390-senior-data-engineer) at **Douglas GmbH**
- [Lead Software Engineer - Data Engineering](https://www.wearedevelopers.com/jobs/ext/2000968-lead-software-engineer-data-engineering) at **Dynatrace**
- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [Staff Business Intelligence Engineer](https://www.wearedevelopers.com/jobs/ext/626164-staff-business-intelligence-engineer) at **Twilio**
- [Senior Software Engineer, Data](https://www.wearedevelopers.com/jobs/48273-senior-software-engineer-data) at **Sportradar Media Services GmbH**
- [Principal Software Engineer, Database Infrastructure](https://www.wearedevelopers.com/jobs/ext/1465908-principal-software-engineer-database-infrastructure) at **GitHub**