> Markdown version of [/videos/73-implementing-continuous-delivery-in-a-data-processing-pipeline?t=694](https://www.wearedevelopers.com/videos/73-implementing-continuous-delivery-in-a-data-processing-pipeline?t=694). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Implementing continuous delivery in a data processing pipeline

Standard continuous delivery practices fail when your final deliverable is data. How can immutable architecture and functional programming principles make your pipeline rollbacks instantaneous?

- **Speakers:** Álvaro Martín Lozano
- **Event:** WeAreDevelopers LIVE
- **Published:** November 12, 2020
- **Duration:** 34:00
- **URL:** https://www.wearedevelopers.com/videos/73-implementing-continuous-delivery-in-a-data-processing-pipeline

## Summary

Transitioning data projects from isolated proofs-of-concept into reliable, production-ready systems requires rethinking traditional continuous delivery. When the final deliverable is raw or transformed data rather than an application binary or a deployed web service, standard deployment practices often fall short. Adapting continuous delivery for data pipelines involves building automated pathways that route code from version control through local Apache Spark integration tests, staging on AWS EMR clusters using test datasets, and final smoke testing before live production release.

To avoid the severe risks and potential corruption associated with complex schema migrations and state modifications, data pipelines can adopt the core principles of functional programming by treating all data as strictly immutable. Each pipeline job acts as a pure function, taking versioned data inputs and configuration parameters to produce a newly versioned, immutable output directory. This architectural shift guarantees idempotent workflows, making the reproduction of historical runs effortless and highly auditable. If a production bug or data anomaly occurs, rolling back is instantaneous—requiring only a pointer change to a previous directory hash rather than executing complex, stateful reverse scripts.

However, functional data engineering introduces unique project orchestration challenges. Standard CI/CD tools often lack the native ability to manage complex dependencies across decoupled jobs where downstream jobs must dynamically link to newly versioned upstream data. Teams must balance building custom deployment tooling to bridge this gap against keeping individual pipeline components asynchronous. While generating fresh datasets for every build sequence exponentially increases storage footprint, leveraging scalable cloud storage and implementing structured lifecycle purges for feature branches makes immutable data architecture a highly resilient standard for modern data operations.

**Keywords:** continuous delivery pipelines, immutable data architectures, functional data engineering, apache spark integration, AWS EMR deployments, idempotent data processing, data versioning strategies, CI/CD orchestration, automated smoke testing, stateless data rollbacks, parquet file versioning, reproducible pipeline execution, production data workflows, staged data deployments

## Chapters

1. **Background on the Data Lab and organizational structure** (00:17) — An overview of research initiatives moving toward production systems across various artificial intelligence domains.
1. **Defining continuous delivery and software deliverables** (04:43) — How deployment artifacts explicitly vary from installed binaries to hosted services and processed data sets.
1. **The path from source control to continuous deployment** (08:57) — Key automation lifecycle stages required to strictly formalize progression to reliable production deployments.
1. **Implementing continuous deployment architectures for data pipelines** (11:34) — Constructing pipelines utilizing isolated local testing, dedicated staging generation, and explicit smoke tests.
1. **Structuring dynamic version labels for automated builds** (14:32) — Using explicit branch identifiers and commit hashes to track execution artifacts before primary code merges.
1. **Generating versioned data artifacts using on-demand clusters** (16:44) — Producing explicitly labeled output directories utilizing dynamically provisioned cloud computing engines across distinct environments.
1. **Replacing stateful migrations with immutable data structures** (18:33) — Applying functional data patterns to explicitly avoid state modifications and natively guarantee job idempotency.
1. **Orchestrating interdependent pipelines across disconnected processing jobs** (22:53) — Addressing the complexity of linking internal dependencies when continuous integration platforms lack dedicated data tooling.
1. **Advantages of reproducible configurations and instantaneous rollbacks** (28:32) — How tracking decoupled immutable states guarantees explicit data auditing and trivial failure recovery.
1. **Mitigating deployment orchestration and excessive storage overhead** (31:49) — Evaluating required operational automations and directory purging procedures needed to balance expanding cloud environments.

## Related Moments

- [Implementing continuous delivery architecture for machine learning](https://www.wearedevelopers.com/videos/178-industrializing-your-data-science-capabilities) (from "Industrializing your Data Science capabilities")
- [Evolution of software delivery and the devops pipeline](https://www.wearedevelopers.com/videos/853-navigating-the-ai-wave-in-devops) (from "Navigating the AI Wave in DevOps")
- [Automating delivery pipelines and managing infrastructure via code](https://www.wearedevelopers.com/videos/545-my-journey-into-devops-world-how-it-all-started) (from "My journey into DevOps world - How it all started!")
- [Applying software engineering environments and testing to data pipelines](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) (from "Modern Data Architectures need Software Engineering")
- [Evolution from manual deployments to continuous integration](https://www.wearedevelopers.com/videos/1603-next-gen-ci-cd-with-gitops-and-progressive-delivery) (from "Next-gen CI/CD with Gitops and Progressive Delivery")
- [Establishing reproducible data science with openness and automation](https://www.wearedevelopers.com/videos/162-data-science-on-software-data) (from "Data Science on Software Data")

## Related Articles

- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)
- [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)
- [Now is the time for industrialized software development](https://www.wearedevelopers.com/magazine/601-now-is-the-time-for-industrialized-software-development)

## Related Jobs

- [Lead Software Engineer - Data Engineering](https://www.wearedevelopers.com/jobs/ext/2000968-lead-software-engineer-data-engineering) at **Dynatrace**
- [Senior Data Engineer](https://www.wearedevelopers.com/jobs/ext/1589390-senior-data-engineer) at **Douglas GmbH**
- [Cloud Data Engineer (all genders)](https://www.wearedevelopers.com/jobs/48216-cloud-data-engineer-all-genders) at **msg**
- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [Devops Engineer](https://www.wearedevelopers.com/jobs/ext/1940926-devops-engineer) at **Bitpanda**
- [Cloud Foundations Team](https://www.wearedevelopers.com/jobs/ext/1483289-cloud-foundations-team) at **GitHub**