WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2023 • Nov 10, 2023

Enjoying SQL data pipelines with dbt

Matthias Niehoff

Transform messy SQL pipelines into scalable software. Learn how dbt applies CI/CD, version control, and modular testing to your analytics, making data engineering strictly code and surprisingly enjoyable.

Pause
Mute Enter Fullscreen
#1 about 3 min

Moving away from unstructured SQL strings

How executing raw commands through scripting prevents pipelines from acting predictably.

#2 about 3 min

Structuring data transformations with the data build tool

How offloading transformation steps into target databases simplifies scaling large pipelines.

#3 about 3 min

Defining data sources and writing preliminary schema tests

How defining raw tables against strict contracts ensures inputs meet baseline assumptions.

#4 about 3 min

Capturing historical state and integrating static reference data

How applying automatic snapshot tracking preserves historically mutable records over time.

#5 about 3 min

Building transformation models with SQL and Jinja macros

How compiling modular jinja templates abstracts away repetitive querying workflows.

#6 about 2 min

Serving documentation and visualizing data lineage automatically

How compiling automated visual graphs exposes exact data movement and dependencies.

#7 about 3 min

Validating data state and utilizing open-source dbt packages

How pulling community packages into pipelines easily applies rigorous structural verifications.

#8 about 3 min

Implementing continuous integration and isolated developer environments

How combining custom schemas with merge checks limits destructive database modifications.

#9 about 4 min

Extending functionality with orchestration and lightweight query engines

How executing transformations against file engines accelerates offline analytical workflows.

#10 about 3 min

Solving data ingestion and recognizing tool boundaries

How delegating extraction responsibilities to specialized tools completes robust engineering architectures.

#11 about 2 min

Handling untyped ingestion and comparing dbt against Spark

How comparing pipeline architectures reveals the operational weight behind large python dependencies.

Matching moments

4:59 min

Bringing DevOps practices to data transformation with DBT

Matthias Niehoff Matthias Niehoff · World Congress 2024

1:49 min

Building the machine learning engineering pipeline

Dora Petrella · World Congress 2023

56 sec

Introduction to analytical data formats for software developers

Matthias Niehoff Matthias Niehoff · World Congress 2026 Europe

2:15 min

Empowering domain teams with an open data platform

Sandhya Menon Sandhya Menon · World Congress 2026 Europe

47 sec

Building modern data pipelines for legacy exports

Dr. Alexander Wachtel Dr. Alexander Wachtel +1 · World Congress 2025

2:57 min

Implementing continuous deployment architectures for data pipelines

Álvaro Martín Lozano · LIVE

More from World Congress 2023

The year 3000, a brief history of Web Development thumbnail
25:30
over 2 years ago

The year 3000, a brief history of Web Development

Lorenzo Pieri

Demografie nutzen um jedem Bewerber das passende Angebot zu machen! thumbnail
8:52
over 2 years ago

Demografie nutzen um jedem Bewerber das passende Angebot zu machen!

Nathalie Gierth

Guided Memory Management: Rust's Ownership Model thumbnail
0:18
almost 3 years ago

Guided Memory Management: Rust's Ownership Model

Stefan Baumgartner

Passwordless future: WebAuthn and Passkeys in practice thumbnail
32:32
almost 3 years ago

Passwordless future: WebAuthn and Passkeys in practice

Clemens Hübner

External Secrets Operator: the secrets management toolbox for self-sufficient teams thumbnail
30:07
almost 3 years ago

External Secrets Operator: the secrets management toolbox for self-sufficient teams

Moritz Johner

Building Reliable Serverless Applications with AWS CDK and Testing thumbnail
25:18
almost 3 years ago

Building Reliable Serverless Applications with AWS CDK and Testing

Raphael Manke

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 25, 2026 · 09:40–10:10

Stage 6

Chat with Your Data: From Natural Language to SQL

Alper Ebicoglu

Co-founder of Volosoft

Alper Ebicoglu
Open session

World Congress 2026 North America

September 25, 2026 · 10:20–10:50

Stage 9

Testing React Backends Like a Pro: Mocking Databases with SQLite

David Morris

Solution Architect

David Morris
Open session

World Congress 2026 North America

September 25, 2026 · 13:30–14:00

Stage 4

From Model Selection to Smart Routing: How to Use the Right LLM for Every Task

Viktoria Semaan

Principal Technical Evangelist at Databricks

Viktoria Semaan
Open session

World Congress 2026 North America

September 25, 2026 · 16:10–16:40

Stage 6

Beyond SQL Generation: How to Teach Agents What Your Database Actually Means

Kris Jenkins

Host / Developer Voices

Kris Jenkins
Open session

World Congress 2026 North America

September 24, 2026 · 16:50–17:20

Stage 9

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

September 24, 2026 · 15:30–16:00

Tech Leaders Stage

Transforming Enterprise AI with Tabular Foundation Models

Leo Grinsztajn, Tom Palczewski, Yaad Oren

Leo Grinsztajn
Tom Palczewski
Yaad Oren

Related videos

Modern Data Architectures need Software Engineering thumbnail
29:25
World Congress 2024 • about 2 years ago

Modern Data Architectures need Software Engineering

Matthias Niehoff

From Messy Queries to Scalable Systems - How Data Engineering actually works thumbnail
31:15
World Congress 2026 Europe • about 2 months ago

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone thumbnail
27:27
World Congress 2026 Europe • about 2 months ago

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone

Daniël van Eeden, Mattias Jonsson

Implementing continuous delivery in a data processing pipeline thumbnail
34:00
LIVE • almost 6 years ago

Implementing continuous delivery in a data processing pipeline

Álvaro Martín Lozano

Database DevOps with Containers thumbnail
39:15
LIVE • over 5 years ago

Database DevOps with Containers

Rob Richardson

OLTP in the Lakehouse: Redefining Data for AI Workloads thumbnail
24:36
Europe 2026 Virtual • 2 months ago

OLTP in the Lakehouse: Redefining Data for AI Workloads

Oleksandra Bovkun

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.