WeAreDevelopers LIVE • Nov 12, 2020

Implementing continuous delivery in a data processing pipeline

Álvaro Martín Lozano

Standard continuous delivery practices fail when your final deliverable is data. How can immutable architecture and functional programming principles make your pipeline rollbacks instantaneous?

Pause

Mute

Enter Fullscreen

#1 about 5 min

Background on the Data Lab and organizational structure

An overview of research initiatives moving toward production systems across various artificial intelligence domains.

#2 about 5 min

Defining continuous delivery and software deliverables

How deployment artifacts explicitly vary from installed binaries to hosted services and processed data sets.

#3 about 3 min

The path from source control to continuous deployment

Key automation lifecycle stages required to strictly formalize progression to reliable production deployments.

#4 about 3 min

Implementing continuous deployment architectures for data pipelines

Constructing pipelines utilizing isolated local testing, dedicated staging generation, and explicit smoke tests.

#5 about 3 min

Structuring dynamic version labels for automated builds

Using explicit branch identifiers and commit hashes to track execution artifacts before primary code merges.

#6 about 2 min

Generating versioned data artifacts using on-demand clusters

Producing explicitly labeled output directories utilizing dynamically provisioned cloud computing engines across distinct environments.

#7 about 5 min

Replacing stateful migrations with immutable data structures

Applying functional data patterns to explicitly avoid state modifications and natively guarantee job idempotency.

#8 about 6 min

Orchestrating interdependent pipelines across disconnected processing jobs

Addressing the complexity of linking internal dependencies when continuous integration platforms lack dedicated data tooling.

#9 about 4 min

Advantages of reproducible configurations and instantaneous rollbacks

How tracking decoupled immutable states guarantees explicit data auditing and trivial failure recovery.

#10 about 3 min

Mitigating deployment orchestration and excessive storage overhead

Evaluating required operational automations and directory purging procedures needed to balance expanding cloud environments.

Matching moments

3:28 min

Implementing continuous delivery architecture for machine learning

Dubravko Dolic +1 · World Congress 2021

7:56 min

Evolution of software delivery and the devops pipeline

Raz Cohen · LIVE

4:29 min

Automating delivery pipelines and managing infrastructure via code

Megha Kadur · LIVE

6:08 min

Applying software engineering environments and testing to data pipelines

Matthias Niehoff

Matthias Niehoff · World Congress 2024

1:38 min

Evolution from manual deployments to continuous integration

Kevin Dubois

Kevin Dubois · World Congress 2025

2:03 min

Establishing reproducible data science with openness and automation

Markus Harrer

Markus Harrer · World Congress 2021

More from WeAreDevelopers LIVE

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along thumbnail

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along

Chris Heilmann, Daniel Cranney, Krzysztof Cieślak

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak thumbnail

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Open Source Agent Toolkits thumbnail

WeAreDevelopers LIVE - Open Source Agent Toolkits

Chris Heilmann, Daniel Cranney, Eddie Jaoude

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude thumbnail

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude

Chris Heilmann, Daniel Cranney

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

WeAreDevelopers - Of Agents and Robots thumbnail

WeAreDevelopers - Of Agents and Robots

Chris Heilmann, Daniel Cranney, Marc Plogas

Upcoming sessions on this topic

World Congress 2026 North America

September 24, 2026 · 10:20–10:50

Tech Leaders Stage

The New Rules of Software Delivery

Darko Mesaros, AJ Aitken, Moritz Plassnig, Akshay Shah

+1 more

Darko Mesaros

AJ Aitken

Moritz Plassnig

Akshay Shah

World Congress 2026 North America

September 24, 2026 · 16:50–17:20

Stage 9

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs

World Congress 2026 North America

September 25, 2026 · 14:50–15:20

Tech Leaders Stage

Keeping Code Quality at AI Speed

Daksh Gupta, Arthur Hicken, Jack Danger

Daksh Gupta

Arthur Hicken

Jack Danger

World Congress 2026 North America

September 25, 2026 · 11:40–12:10

Stage 9

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan

World Congress 2026 North America

September 24, 2026 · 11:10–11:15

Outdoor Stage

Architecting the 100X SDLC: Building Production Trust into AI-Assisted Delivery

Ranjan Parthasarathy

Founder, CPTO/CEO at AXIOMSTUDIO.AI

Ranjan Parthasarathy

World Congress 2026 North America

September 24, 2026 · 17:30–18:00

Stage 4

Boring Failover: Predictable Region Recovery Across 5,000 Microservices

Garvit Kataria, Sahil Sabharwal

Garvit Kataria

Sahil Sabharwal

Related videos

Modern Data Architectures need Software Engineering thumbnail

World Congress 2024 • about 2 years ago

Modern Data Architectures need Software Engineering

Matthias Niehoff

Integrating backups into your GitOps Pipeline thumbnail

World Congress 2023 • almost 3 years ago

Integrating backups into your GitOps Pipeline

Florian Trieloff

From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform thumbnail

World Congress 2026 Europe • 2 months ago

From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform

Dominik Krichbaum

Plan CI/CD on the Enterprise level! thumbnail

LIVE • over 3 years ago

Plan CI/CD on the Enterprise level!

Pawel Piwosz

From Messy Queries to Scalable Systems - How Data Engineering actually works thumbnail

World Congress 2026 Europe • 2 months ago

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

Industrializing your Data Science capabilities thumbnail

World Congress 2021 • about 5 years ago

Industrializing your Data Science capabilities

Dubravko Dolic, Hüdaverdi Cakir

Related articles

BB

Benedikt Bischof

MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production

Welcome to this issue of the WeAreDevelopers Live Talk series. This article recaps an interesting talk by Bas Geerdink who gave advice on MLOps.‍About the speaker:‍Bas is a programmer, scientist, a...

about 4 years ago

MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production

BB

Benedikt Bischof

MLOps – What’s the deal behind it?

Welcome to this issue of the WeAreDevelopers Live Talk series. This article recaps an interesting talk by Nico Axtmann who introduced us to MLOpsAbout the speaker:Nico Axtmann is a seasoned machine...

almost 4 years ago

MLOps – What’s the deal behind it?

BR

Benjamin Ruschin

Navigating the AI Shift

AI has had an undeniable impact on all kinds of aspects of life and work, from how we do everyday tasks, to how software is built, how companies operate, and even how work itself is defined. Despit...

12 months ago

A laptop interacting with AI

AB

Andre Braun, GitLab

Now is the time for industrialized software development

Now is the time for industrialized software development Recently, I received a letter from my car’s manufacturer alerting me to a recall. They had discovered a defective part and wanted to replace ...

about 1 year ago

A person developint HTML on a laptop with the sun shining on it.

From learning to earning

Jobs that call for the skills explored in this talk.

Data Engineer

Berlin, Germany

Docker Airflow Backend

about 1 month ago

Lead Software Engineer - Data Engineering

Barcelona, Spain

NoSQL DevOps Docker

Data Engineer - Manager

Experienced Github Fastapi Pytorch

Data Engineer - Manager

Experienced Github Fastapi Pytorch

Data Engineer - Manager

Experienced Github Fastapi Pytorch

Data Engineer (all genders welcome)

Wietmarschen, Germany

Remote Git Docker Power BI