WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Nov 12, 2020

Implementing continuous delivery in a data processing pipeline

Álvaro Martín Lozano

Standard continuous delivery practices fail when your final deliverable is data. How can immutable architecture and functional programming principles make your pipeline rollbacks instantaneous?

Pause
Mute Enter Fullscreen
#1 about 5 min

Background on the Data Lab and organizational structure

An overview of research initiatives moving toward production systems across various artificial intelligence domains.

#2 about 5 min

Defining continuous delivery and software deliverables

How deployment artifacts explicitly vary from installed binaries to hosted services and processed data sets.

#3 about 3 min

The path from source control to continuous deployment

Key automation lifecycle stages required to strictly formalize progression to reliable production deployments.

#4 about 3 min

Implementing continuous deployment architectures for data pipelines

Constructing pipelines utilizing isolated local testing, dedicated staging generation, and explicit smoke tests.

#5 about 3 min

Structuring dynamic version labels for automated builds

Using explicit branch identifiers and commit hashes to track execution artifacts before primary code merges.

#6 about 2 min

Generating versioned data artifacts using on-demand clusters

Producing explicitly labeled output directories utilizing dynamically provisioned cloud computing engines across distinct environments.

#7 about 5 min

Replacing stateful migrations with immutable data structures

Applying functional data patterns to explicitly avoid state modifications and natively guarantee job idempotency.

#8 about 6 min

Orchestrating interdependent pipelines across disconnected processing jobs

Addressing the complexity of linking internal dependencies when continuous integration platforms lack dedicated data tooling.

#9 about 4 min

Advantages of reproducible configurations and instantaneous rollbacks

How tracking decoupled immutable states guarantees explicit data auditing and trivial failure recovery.

#10 about 3 min

Mitigating deployment orchestration and excessive storage overhead

Evaluating required operational automations and directory purging procedures needed to balance expanding cloud environments.

Matching moments

3:28 min

Implementing continuous delivery architecture for machine learning

Dubravko Dolic +1 · WWC 2021

7:56 min

Evolution of software delivery and the devops pipeline

Raz Cohen · LIVE

4:29 min

Automating delivery pipelines and managing infrastructure via code

Megha Kadur · LIVE

6:08 min

Applying software engineering environments and testing to data pipelines

Matthias Niehoff Matthias Niehoff · WWC 2024

1:38 min

Evolution from manual deployments to continuous integration

Kevin Dubois Kevin Dubois · WWC 2025

2:03 min

Establishing reproducible data science with openness and automation

Markus Harrer Markus Harrer · WWC 2021

More from WeAreDevelopers LIVE

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
5 days ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Node and Package Security thumbnail
1:08:30
5 days ago

WeAreDevelopers LIVE - Node and Package Security

Chris Heilmann, Daniel Cranney, Zbyszek Tenerowicz

WeAreDevelopers LIVE - Streaming HTML thumbnail
1:10:26
12 days ago

WeAreDevelopers LIVE - Streaming HTML

Chris Heilmann, Daniel Cranney, Jad Joubran

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe thumbnail
1:02:34
19 days ago

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe

Chris Heilmann, Daniel Cranney, Justin Halsall, Werner Jeschko, Nicole Jeschko, Mirco Tomić

WeAreDevelopers LIVE - Web Performance in an AI World thumbnail
1:07:59
26 days ago

WeAreDevelopers LIVE - Web Performance in an AI World

Chris Heilmann, Daniel Cranney

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko thumbnail
5:55
about 2 months ago

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

World Congress 2026 North America

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan
Open session

World Congress 2026 North America

Boring Failover: Predictable Region Recovery Across 5,000 Microservices

Garvit Kataria, Sahil Sabharwal

Garvit Kataria
Sahil Sabharwal
Open session

World Congress 2026 North America

It’s Alive! Taming the MLOps Franken-Stack: Write, Run, and Serve with Michelangelo

Paul Zimmerman, Eric Wang

Paul Zimmerman
Eric Wang
Open session

World Congress 2026 North America

Reinventing Testing Practices in the AI Era

Eric Deandrea

Java Champion & Senior Principal Software Engineer, IBM

Eric Deandrea
Open session

World Congress 2026 North America

The Broken Rung: How AI is Rebuilding Software Development from the Ground Up

Tomislav Tipurić

Chief Technology Officer, Nephos

Tomislav Tipurić

Related videos

Modern Data Architectures need Software Engineering thumbnail
29:25
WWC 2024 • almost 2 years ago

Modern Data Architectures need Software Engineering

Matthias Niehoff

Integrating backups into your GitOps Pipeline thumbnail
25:24
WWC 2023 • almost 3 years ago

Integrating backups into your GitOps Pipeline

Florian Trieloff

From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform thumbnail
22:57
WWC Europe 2026 • about 1 month ago

From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform

Dominik Krichbaum

Plan CI/CD on the Enterprise level! thumbnail
56:19
LIVE • over 3 years ago

Plan CI/CD on the Enterprise level!

Pawel Piwosz

From Messy Queries to Scalable Systems - How Data Engineering actually works thumbnail
31:15
WWC Europe 2026 • about 1 month ago

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

Industrializing your Data Science capabilities thumbnail
43:47
WWC 2021 • about 5 years ago

Industrializing your Data Science capabilities

Dubravko Dolic, Hüdaverdi Cakir

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.