> Markdown version of [/videos/823-enjoying-sql-data-pipelines-with-dbt?t=975](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt?t=975). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Enjoying SQL data pipelines with dbt

Transform messy SQL pipelines into scalable software. Learn how dbt applies CI/CD, version control, and modular testing to your analytics, making data engineering strictly code and surprisingly enjoyable.

- **Speakers:** [Matthias Niehoff](https://www.wearedevelopers.com/@matthias-niehoff)
- **Event:** World Congress 2023
- **Published:** November 10, 2023
- **Duration:** 26:53
- **URL:** https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt

## Summary

Historically, software developers tend to avoid SQL-based ETL workflows due to their lack of standardization, reliance on messy string manipulation within Jupyter notebooks, and absence of rigorous version control. Enter dbt (data build tool), a framework that revitalizes SQL data pipelines by treating data transformations strictly as code. By focusing exclusively on the transform phase in the ELT architecture, dbt allows teams to apply standard software engineering methodologies—such as CI/CD, isolated developer sandboxes, and pre-commit hooks—directly to their modern cloud data warehouses. This ensures that analytical logic remains scalable, modular, and testable without requiring heavy compute frameworks like Apache Spark for purely SQL-based tasks. The power of dbt lies in its fusion of standard SQL with Jinja templating and YAML configurations. This combination enables engineers to define robust data models, automate the snapshotting of historical table states, and build deeply reusable macros. Furthermore, dbt acts as a powerful governance mechanism by automatically generating lightweight, web-based data catalogs complete with visual data lineage graphs. Instead of manually tracking how metrics are derived, teams can visually inspect how raw sources flow into staging and production models. This built-in documentation pairs perfectly with dbt's native generic testing and package extensions, providing an integrated feedback loop before any modified table merges into production. While dbt fundamentally changes how analytical schemas are built, it is carefully designed to integrate seamlessly into a broader data engineering ecosystem. Engineers typically manage orchestration by containerizing their dbt runs via Docker or triggering them through tools like Airflow, while leaving data ingestion entirely to specialized platforms like Airbyte or Fivetran. The tool also opens entirely new use cases for developers by combining neatly with Lightdash for direct BI visualization or leveraging DuckDB for lightning-fast analytical queries against raw Parquet files over object storage. Ultimately, dbt proves that when equipped with the right testing and modularity structures, SQL remains an incredibly enjoyable language for modern analytics.

**Keywords:** dbt pipeline architecture, elt data transformation, sql software engineering, jinja sql templates, dbt table snapshots, visual data lineage, lightweight data catalogs, automated pipeline testing, isolated developer schemas, duckdb parquet queries, ci/cd data integration, airflow orchestration, snowflake native transformations, apache spark alternative

## Chapters

1. **Moving away from unstructured SQL strings** (00:03) — How executing raw commands through scripting prevents pipelines from acting predictably.
1. **Structuring data transformations with the data build tool** (02:41) — How offloading transformation steps into target databases simplifies scaling large pipelines.
1. **Defining data sources and writing preliminary schema tests** (04:54) — How defining raw tables against strict contracts ensures inputs meet baseline assumptions.
1. **Capturing historical state and integrating static reference data** (07:05) — How applying automatic snapshot tracking preserves historically mutable records over time.
1. **Building transformation models with SQL and Jinja macros** (09:37) — How compiling modular jinja templates abstracts away repetitive querying workflows.
1. **Serving documentation and visualizing data lineage automatically** (12:25) — How compiling automated visual graphs exposes exact data movement and dependencies.
1. **Validating data state and utilizing open-source dbt packages** (14:05) — How pulling community packages into pipelines easily applies rigorous structural verifications.
1. **Implementing continuous integration and isolated developer environments** (16:15) — How combining custom schemas with merge checks limits destructive database modifications.
1. **Extending functionality with orchestration and lightweight query engines** (18:41) — How executing transformations against file engines accelerates offline analytical workflows.
1. **Solving data ingestion and recognizing tool boundaries** (22:21) — How delegating extraction responsibilities to specialized tools completes robust engineering architectures.
1. **Handling untyped ingestion and comparing dbt against Spark** (25:20) — How comparing pipeline architectures reveals the operational weight behind large python dependencies.

## Related Moments

- [Bringing DevOps practices to data transformation with DBT](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) (from "Modern Data Architectures need Software Engineering")
- [Building the machine learning engineering pipeline](https://www.wearedevelopers.com/videos/752-how-we-built-a-machine-learning-based-recommendation-system-and-survived-to-tell-the-tale) (from "How We Built a Machine Learning-Based Recommendation System (And Survived to Tell the Tale)")
- [Introduction to analytical data formats for software developers](https://www.wearedevelopers.com/videos/100075-parquet-delta-iceberg-ducklake-an-introduction-for-developers) (from "Parquet, Delta, Iceberg & Ducklake - An introduction for developers")
- [Empowering domain teams with an open data platform](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) (from "From Messy Queries to Scalable Systems - How Data Engineering actually works")
- [Building modern data pipelines for legacy exports](https://www.wearedevelopers.com/videos/1547-data-analytics-with-microsoft-fabric-end-to-end-use-case-with-data-agents) (from "Data Analytics with Microsoft Fabric: End-to-End Use Case with Data Agents")
- [Implementing continuous deployment architectures for data pipelines](https://www.wearedevelopers.com/videos/73-implementing-continuous-delivery-in-a-data-processing-pipeline) (from "Implementing continuous delivery in a data processing pipeline")

## Related Articles

- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix)
- [Dev Digest 159: AI Pipelines, 10x Faster TypeScript, How to Interview](https://www.wearedevelopers.com/magazine/563-dev-digest-159-ai-pipelines-10x-faster-typescript-how-to-interview)

## Related Jobs

- [Senior Data Engineer](https://www.wearedevelopers.com/jobs/ext/1589390-senior-data-engineer) at **Douglas GmbH**
- [Lead Software Engineer - Data Engineering](https://www.wearedevelopers.com/jobs/ext/2000968-lead-software-engineer-data-engineering) at **Dynatrace**
- [Staff Business Intelligence Engineer](https://www.wearedevelopers.com/jobs/ext/626164-staff-business-intelligence-engineer) at **Twilio**
- [Staff, Analytics Engineer](https://www.wearedevelopers.com/jobs/ext/569504-staff-analytics-engineer) at **Twilio**
- [Staff, Business Intelligence Engineer](https://www.wearedevelopers.com/jobs/ext/1401813-staff-business-intelligence-engineer) at **Twilio**
- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**