WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Nov 12, 2020

Implementing continuous delivery in a data processing pipeline

Álvaro Martín Lozano

Standard continuous delivery practices fail when your final deliverable is data. How can immutable architecture and functional programming principles make your pipeline rollbacks instantaneous?

Pause
Mute Enter Fullscreen
#1 about 5 min

Background on the Data Lab and organizational structure

An overview of research initiatives moving toward production systems across various artificial intelligence domains.

#2 about 5 min

Defining continuous delivery and software deliverables

How deployment artifacts explicitly vary from installed binaries to hosted services and processed data sets.

#3 about 3 min

The path from source control to continuous deployment

Key automation lifecycle stages required to strictly formalize progression to reliable production deployments.

#4 about 3 min

Implementing continuous deployment architectures for data pipelines

Constructing pipelines utilizing isolated local testing, dedicated staging generation, and explicit smoke tests.

#5 about 3 min

Structuring dynamic version labels for automated builds

Using explicit branch identifiers and commit hashes to track execution artifacts before primary code merges.

#6 about 2 min

Generating versioned data artifacts using on-demand clusters

Producing explicitly labeled output directories utilizing dynamically provisioned cloud computing engines across distinct environments.

#7 about 5 min

Replacing stateful migrations with immutable data structures

Applying functional data patterns to explicitly avoid state modifications and natively guarantee job idempotency.

#8 about 6 min

Orchestrating interdependent pipelines across disconnected processing jobs

Addressing the complexity of linking internal dependencies when continuous integration platforms lack dedicated data tooling.

#9 about 4 min

Advantages of reproducible configurations and instantaneous rollbacks

How tracking decoupled immutable states guarantees explicit data auditing and trivial failure recovery.

#10 about 3 min

Mitigating deployment orchestration and excessive storage overhead

Evaluating required operational automations and directory purging procedures needed to balance expanding cloud environments.

Matching moments

3:28 min

Implementing continuous delivery architecture for machine learning

Dubravko Dolic +1 · World Congress 2021

7:56 min

Evolution of software delivery and the devops pipeline

Raz Cohen · LIVE

4:29 min

Automating delivery pipelines and managing infrastructure via code

Megha Kadur · LIVE

6:08 min

Applying software engineering environments and testing to data pipelines

Matthias Niehoff Matthias Niehoff · World Congress 2024

1:38 min

Evolution from manual deployments to continuous integration

Kevin Dubois Kevin Dubois · World Congress 2025

2:03 min

Establishing reproducible data science with openness and automation

Markus Harrer Markus Harrer · World Congress 2021

More from WeAreDevelopers LIVE

WeAreDevelopers LIVE - Is CSS Catching Up to Flash? thumbnail
1:03:33
12 days ago

WeAreDevelopers LIVE - Is CSS Catching Up to Flash?

Chris Heilmann, Daniel Cranney, Bramus Van Damme

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along thumbnail
1:21:17
19 days ago

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along

Chris Heilmann, Daniel Cranney, Krzysztof Cieślak

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak thumbnail
4:23
21 days ago

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Open Source Agent Toolkits thumbnail
1:00:38
26 days ago

WeAreDevelopers LIVE - Open Source Agent Toolkits

Chris Heilmann, Daniel Cranney, Eddie Jaoude

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude thumbnail
6:16
28 days ago

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude

Chris Heilmann, Daniel Cranney

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail
5:46
about 1 month ago

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

Related videos

Modern Data Architectures need Software Engineering thumbnail
29:25
World Congress 2024 • about 2 years ago

Modern Data Architectures need Software Engineering

Matthias Niehoff

Integrating backups into your GitOps Pipeline thumbnail
25:24
World Congress 2023 • almost 3 years ago

Integrating backups into your GitOps Pipeline

Florian Trieloff

From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform thumbnail
22:57
World Congress 2026 Europe • 3 months ago

From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform

Dominik Krichbaum

Plan CI/CD on the Enterprise level! thumbnail
56:19
LIVE • over 3 years ago

Plan CI/CD on the Enterprise level!

Pawel Piwosz

From Messy Queries to Scalable Systems - How Data Engineering actually works thumbnail
31:15
World Congress 2026 Europe • 3 months ago

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

Industrializing your Data Science capabilities thumbnail
43:47
World Congress 2021 • over 5 years ago

Industrializing your Data Science capabilities

Dubravko Dolic, Hüdaverdi Cakir

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.