WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2023 • Nov 10, 2023

Enjoying SQL data pipelines with dbt

Matthias Niehoff

Transform messy SQL pipelines into scalable software. Learn how dbt applies CI/CD, version control, and modular testing to your analytics, making data engineering strictly code and surprisingly enjoyable.

Pause
Mute Enter Fullscreen
#1 about 3 min

Moving away from unstructured SQL strings

How executing raw commands through scripting prevents pipelines from acting predictably.

#2 about 3 min

Structuring data transformations with the data build tool

How offloading transformation steps into target databases simplifies scaling large pipelines.

#3 about 3 min

Defining data sources and writing preliminary schema tests

How defining raw tables against strict contracts ensures inputs meet baseline assumptions.

#4 about 3 min

Capturing historical state and integrating static reference data

How applying automatic snapshot tracking preserves historically mutable records over time.

#5 about 3 min

Building transformation models with SQL and Jinja macros

How compiling modular jinja templates abstracts away repetitive querying workflows.

#6 about 2 min

Serving documentation and visualizing data lineage automatically

How compiling automated visual graphs exposes exact data movement and dependencies.

#7 about 3 min

Validating data state and utilizing open-source dbt packages

How pulling community packages into pipelines easily applies rigorous structural verifications.

#8 about 3 min

Implementing continuous integration and isolated developer environments

How combining custom schemas with merge checks limits destructive database modifications.

#9 about 4 min

Extending functionality with orchestration and lightweight query engines

How executing transformations against file engines accelerates offline analytical workflows.

#10 about 3 min

Solving data ingestion and recognizing tool boundaries

How delegating extraction responsibilities to specialized tools completes robust engineering architectures.

#11 about 2 min

Handling untyped ingestion and comparing dbt against Spark

How comparing pipeline architectures reveals the operational weight behind large python dependencies.

Matching moments

4:59 min

Bringing DevOps practices to data transformation with DBT

Matthias Niehoff Matthias Niehoff · WWC 2024

1:49 min

Building the machine learning engineering pipeline

Dora Petrella · WWC 2023

56 sec

Introduction to analytical data formats for software developers

Matthias Niehoff Matthias Niehoff · WWC Europe 2026

2:15 min

Empowering domain teams with an open data platform

Sandhya Menon Sandhya Menon · WWC Europe 2026

47 sec

Building modern data pipelines for legacy exports

Dr. Alexander Wachtel Dr. Alexander Wachtel +1 · WWC 2025

2:57 min

Implementing continuous deployment architectures for data pipelines

Álvaro Martín Lozano · LIVE

More from World Congress 2023

The year 3000, a brief history of Web Development thumbnail
25:30
over 2 years ago

The year 3000, a brief history of Web Development

Lorenzo Pieri

Demografie nutzen um jedem Bewerber das passende Angebot zu machen! thumbnail
8:52
over 2 years ago

Demografie nutzen um jedem Bewerber das passende Angebot zu machen!

Nathalie Gierth

Guided Memory Management: Rust's Ownership Model thumbnail
0:18
almost 3 years ago

Guided Memory Management: Rust's Ownership Model

Stefan Baumgartner

Passwordless future: WebAuthn and Passkeys in practice thumbnail
32:32
almost 3 years ago

Passwordless future: WebAuthn and Passkeys in practice

Clemens Hübner

External Secrets Operator: the secrets management toolbox for self-sufficient teams thumbnail
30:07
almost 3 years ago

External Secrets Operator: the secrets management toolbox for self-sufficient teams

Moritz Johner

Building Reliable Serverless Applications with AWS CDK and Testing thumbnail
25:18
almost 3 years ago

Building Reliable Serverless Applications with AWS CDK and Testing

Raphael Manke

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Chat with Your Data: From Natural Language to SQL

Alper Ebicoglu

Co-founder of Volosoft

Alper Ebicoglu
Open session

World Congress 2026 North America

Testing React Backends Like a Pro: Mocking Databases with SQLite

David Morris

Solution Architect

David Morris
Open session

World Congress 2026 North America

Ship 10x Faster: AI-Powered Development with Claude Code and MCP Tools

Viktoria Semaan

Principal Technical Evangelist at Databricks

Viktoria Semaan
Open session

World Congress 2026 North America

Beyond SQL Generation: How to Teach Agents What Your Database Actually Means

Kris Jenkins

Lead Developer Advocate at Snowflake

Kris Jenkins
Open session

World Congress 2026 North America

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

AI Agents are Only as Smart as their Context: Building a Real-Time Context Engine at Intuit

Bharat Patel

Lead Software Engineer at Intuit

Bharat Patel

Related videos

Modern Data Architectures need Software Engineering thumbnail
29:25
WWC 2024 • almost 2 years ago

Modern Data Architectures need Software Engineering

Matthias Niehoff

From Messy Queries to Scalable Systems - How Data Engineering actually works thumbnail
31:15
WWC Europe 2026 • about 1 month ago

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone thumbnail
27:27
WWC Europe 2026 • about 1 month ago

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone

Daniël van Eeden, Mattias Jonsson

Implementing continuous delivery in a data processing pipeline thumbnail
34:00
LIVE • over 5 years ago

Implementing continuous delivery in a data processing pipeline

Álvaro Martín Lozano

Database DevOps with Containers thumbnail
39:15
LIVE • about 5 years ago

Database DevOps with Containers

Rob Richardson

Parquet, Delta, Iceberg & Ducklake - An introduction for developers thumbnail
28:42
WWC Europe 2026 • about 1 month ago

Parquet, Delta, Iceberg & Ducklake - An introduction for developers

Matthias Niehoff

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.