Álvaro Martín Lozano
Implementing continuous delivery in a data processing pipeline
#1about 4 minutes
From research concepts to production-ready data products
The Volkswagen Data Lab shifted its focus from demonstrating proof-of-concepts to building and deploying real-world data solutions for its clients.
#2about 7 minutes
Core concepts of continuous delivery for data
Continuous delivery for data pipelines requires adapting standard CI/CD principles, where data is the deliverable, by progressing through version control, integration, and deployment stages.
#3about 11 minutes
Implementing a pipeline with immutable, versioned data
The five-step pipeline relies on treating data as immutable, creating a new versioned output for each run to enable simple rollbacks and reproducibility.
#4about 6 minutes
The challenge of orchestrating chained data jobs
Managing dependencies between jobs becomes complex when each job consumes versioned, immutable data inputs from upstream processes.
#5about 5 minutes
Pros and cons of the immutable data approach
While this method offers powerful benefits like reproducibility and instant rollbacks, it introduces challenges in orchestration complexity and increased storage costs.
Related jobs
Jobs that call for the skills explored in this talk.
ROSEN Technology and Research Center GmbH
Osnabrück, Germany
Senior
TypeScript
React
+3
Matching moments
04:57 MIN
Increasing the value of talk recordings post-event
Cat Herding with Lions and Tigers - Christian Heilmann
02:39 MIN
Establishing a single source of truth for all data
Cat Herding with Lions and Tigers - Christian Heilmann
01:32 MIN
Organizing a developer conference for 15,000 attendees
Cat Herding with Lions and Tigers - Christian Heilmann
02:54 MIN
Automating video post-production with local scripts
Cat Herding with Lions and Tigers - Christian Heilmann
03:28 MIN
Why corporate AI adoption lags behind the hype
What 2025 Taught Us: A Year-End Special with Hung Lee
04:22 MIN
Why HR struggles with technology implementation and adoption
What 2025 Taught Us: A Year-End Special with Hung Lee
03:39 MIN
Breaking down silos between HR, tech, and business
What 2025 Taught Us: A Year-End Special with Hung Lee
03:48 MIN
Automating formal processes risks losing informal human value
What 2025 Taught Us: A Year-End Special with Hung Lee
Featured Partners
Related Videos
39:04Python-Based Data Streaming Pipelines Within Minutes
Bobur Umurzokov
30:46Charting the Journey to Continuous Deployment with a Value Stream Map
Josh Armitage
43:47Industrializing your Data Science capabilities
Dubravko Dolic & Hüdaverdi Cakir
44:40Enabling automated 1-click customer deployments with built-in quality and security
Christoph Ruggenthaler
27:52Progressive Delivery in Kubernetes
Carlos Sanchez
29:36CI/CD Patterns and Antipatterns - Things your Pipeline Should (Not) Do
Daniel Raniz Raneland
50:28Practical tips and tricks for CI/CD success
Zan Markan
29:25Modern Data Architectures need Software Engineering
Matthias Niehoff
Related Articles
View all articles
From learning to earning
Jobs that call for the skills explored in this talk.
Tenth Revolution
Barcelona, Spain
Intermediate
API
PySpark
Amazon Web Services (AWS)
Crossing Hurdles
Municipality of Madrid, Spain
Remote
Python
Machine Learning
mund consulting AG
Berlin, Germany
Intermediate
ETL
Spark
Python
Gitlab
Confluence
+2
Frontiers
Municipality of Madrid, Spain
Intermediate
osapiens
Municipality of Madrid, Spain
Intermediate
Azure
Continuous Integration