WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2023 • Nov 10, 2023

Enjoying SQL data pipelines with dbt

Matthias Niehoff

Transform messy SQL pipelines into scalable software. Learn how dbt applies CI/CD, version control, and modular testing to your analytics, making data engineering strictly code and surprisingly enjoyable.

Pause
Mute Enter Fullscreen
#1 about 3 min

Moving away from unstructured SQL strings

How executing raw commands through scripting prevents pipelines from acting predictably.

#2 about 3 min

Structuring data transformations with the data build tool

How offloading transformation steps into target databases simplifies scaling large pipelines.

#3 about 3 min

Defining data sources and writing preliminary schema tests

How defining raw tables against strict contracts ensures inputs meet baseline assumptions.

#4 about 3 min

Capturing historical state and integrating static reference data

How applying automatic snapshot tracking preserves historically mutable records over time.

#5 about 3 min

Building transformation models with SQL and Jinja macros

How compiling modular jinja templates abstracts away repetitive querying workflows.

#6 about 2 min

Serving documentation and visualizing data lineage automatically

How compiling automated visual graphs exposes exact data movement and dependencies.

#7 about 3 min

Validating data state and utilizing open-source dbt packages

How pulling community packages into pipelines easily applies rigorous structural verifications.

#8 about 3 min

Implementing continuous integration and isolated developer environments

How combining custom schemas with merge checks limits destructive database modifications.

#9 about 4 min

Extending functionality with orchestration and lightweight query engines

How executing transformations against file engines accelerates offline analytical workflows.

#10 about 3 min

Solving data ingestion and recognizing tool boundaries

How delegating extraction responsibilities to specialized tools completes robust engineering architectures.

#11 about 2 min

Handling untyped ingestion and comparing dbt against Spark

How comparing pipeline architectures reveals the operational weight behind large python dependencies.

Matching moments

4:59 min

Bringing DevOps practices to data transformation with DBT

Matthias Niehoff Matthias Niehoff · World Congress 2024

1:49 min

Building the machine learning engineering pipeline

Dora Petrella · World Congress 2023

56 sec

Introduction to analytical data formats for software developers

Matthias Niehoff Matthias Niehoff · World Congress 2026 Europe

2:15 min

Empowering domain teams with an open data platform

Sandhya Menon Sandhya Menon · World Congress 2026 Europe

47 sec

Building modern data pipelines for legacy exports

Dr. Alexander Wachtel Dr. Alexander Wachtel +1 · World Congress 2025

2:57 min

Implementing continuous deployment architectures for data pipelines

Álvaro Martín Lozano · LIVE

More from World Congress 2023

The year 3000, a brief history of Web Development thumbnail
25:30
over 2 years ago

The year 3000, a brief history of Web Development

Lorenzo Pieri

Demografie nutzen um jedem Bewerber das passende Angebot zu machen! thumbnail
8:52
almost 3 years ago

Demografie nutzen um jedem Bewerber das passende Angebot zu machen!

Nathalie Gierth

Guided Memory Management: Rust's Ownership Model thumbnail
0:18
almost 3 years ago

Guided Memory Management: Rust's Ownership Model

Stefan Baumgartner

Passwordless future: WebAuthn and Passkeys in practice thumbnail
32:32
almost 3 years ago

Passwordless future: WebAuthn and Passkeys in practice

Clemens Hübner

External Secrets Operator: the secrets management toolbox for self-sufficient teams thumbnail
30:07
almost 3 years ago

External Secrets Operator: the secrets management toolbox for self-sufficient teams

Moritz Johner

Building Reliable Serverless Applications with AWS CDK and Testing thumbnail
25:18
almost 3 years ago

Building Reliable Serverless Applications with AWS CDK and Testing

Raphael Manke

Related videos

Modern Data Architectures need Software Engineering thumbnail
29:25
World Congress 2024 • about 2 years ago

Modern Data Architectures need Software Engineering

Matthias Niehoff

From Messy Queries to Scalable Systems - How Data Engineering actually works thumbnail
31:15
World Congress 2026 Europe • 3 months ago

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone thumbnail
27:27
World Congress 2026 Europe • 3 months ago

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone

Daniël van Eeden, Mattias Jonsson

Implementing continuous delivery in a data processing pipeline thumbnail
34:00
LIVE • almost 6 years ago

Implementing continuous delivery in a data processing pipeline

Álvaro Martín Lozano

Database DevOps with Containers thumbnail
39:15
LIVE • over 5 years ago

Database DevOps with Containers

Rob Richardson

OLTP in the Lakehouse: Redefining Data for AI Workloads thumbnail
24:36
Europe 2026 Virtual • 3 months ago

OLTP in the Lakehouse: Redefining Data for AI Workloads

Oleksandra Bovkun

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.