Álvaro Martín Lozano

Implementing continuous delivery in a data processing pipeline

What if you could roll back a data pipeline instantly, with zero state migration? See how treating data as an immutable, versioned artifact makes this a reality.

Implementing continuous delivery in a data processing pipeline
#1about 4 minutes

From research concepts to production-ready data products

The Volkswagen Data Lab shifted its focus from demonstrating proof-of-concepts to building and deploying real-world data solutions for its clients.

#2about 7 minutes

Core concepts of continuous delivery for data

Continuous delivery for data pipelines requires adapting standard CI/CD principles, where data is the deliverable, by progressing through version control, integration, and deployment stages.

#3about 11 minutes

Implementing a pipeline with immutable, versioned data

The five-step pipeline relies on treating data as immutable, creating a new versioned output for each run to enable simple rollbacks and reproducibility.

#4about 6 minutes

The challenge of orchestrating chained data jobs

Managing dependencies between jobs becomes complex when each job consumes versioned, immutable data inputs from upstream processes.

#5about 5 minutes

Pros and cons of the immutable data approach

While this method offers powerful benefits like reproducibility and instant rollbacks, it introduces challenges in orchestration complexity and increased storage costs.

Related jobs
Jobs that call for the skills explored in this talk.

Matching moments

The distinct roles of CI and CD pipelines
00:36 MIN

The distinct roles of CI and CD pipelines

#90DaysOfDevOps - The DevOps Learning Journey

Using continuous delivery to enable business agility
06:04 MIN

Using continuous delivery to enable business agility

The Affordances of Quality

Defining continuous integration, delivery, and deployment
00:51 MIN

Defining continuous integration, delivery, and deployment

CI/CD with Github Actions

Tracing the evolution of DevOps from silos to superhighways
05:34 MIN

Tracing the evolution of DevOps from silos to superhighways

Navigating the AI Wave in DevOps

Implementing an AI-in-the-loop continuous learning cycle
09:45 MIN

Implementing an AI-in-the-loop continuous learning cycle

A solution to embed container technologies into automotive environments

Core principles for achieving continuous deployment
01:57 MIN

Core principles for achieving continuous deployment

Charting the Journey to Continuous Deployment with a Value Stream Map

Moving beyond tools to architect CI/CD processes
03:49 MIN

Moving beyond tools to architect CI/CD processes

Plan CI/CD on the Enterprise level!

Differentiating continuous delivery and continuous deployment
03:31 MIN

Differentiating continuous delivery and continuous deployment

Plan CI/CD on the Enterprise level!

Featured Partners

Related Videos

See all videos
Charting the Journey to Continuous Deployment with a Value Stream Map30:46

Charting the Journey to Continuous Deployment with a Value Stream Map

Josh Armitage

Industrializing your Data Science capabilities43:47

Industrializing your Data Science capabilities

Dubravko Dolic & Hüdaverdi Cakir

Plan CI/CD on the Enterprise level!56:19

Plan CI/CD on the Enterprise level!

Pawel Piwosz

DevOps for Machine Learning44:40

DevOps for Machine Learning

Hauke Brammer

Integrating backups into your GitOps Pipeline25:24

Integrating backups into your GitOps Pipeline

Florian Trieloff

CD2CF - Continuous Deployment to Cloud Foundry46:06

CD2CF - Continuous Deployment to Cloud Foundry

Dominik Kress

Database DevOps with Containers39:15

Database DevOps with Containers

Rob Richardson

Navigating the AI Wave in DevOps54:46

Navigating the AI Wave in DevOps

Raz Cohen

Related Articles

View all articles
AG
Andre Braun, GitLab
Now is the time for industrialized software development
Now is the time for industrialized software development Recently, I received a letter from my car’s manufacturer alerting me to a recall. They had discovered a defective part and wanted to replace it. It was easily fixed, and I might have forgotten a...
Now is the time for industrialized software development
DC
Daniel Cranney
Why Attend a Developer Event?
Modern software engineering moves too fast for documentation alone. Attending a world-class event is about shifting from tactical execution to strategic leadership. Skill Diversification: Break out of your specific tech stack to see how the industry...
Why Attend a Developer Event?

From learning to earning

Jobs that call for the skills explored in this talk.