> Markdown version of [/jobs/ext/4088990-cdi-data-engineer](https://www.wearedevelopers.com/jobs/ext/4088990-cdi-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# CDI Data Engineer

- **Company:** Michael Page
- **Location:** London, United Kingdom
- **Salary:** £93,600 - £114,400
- **Contract:** Temporary contract
- **Skills:** Application Programming Interfaces (APIs), Amazon Web Services, Unit Testing, Microsoft Azure, Continuous Integration, Data Validation, Information Engineering, Extract Transform Load (ETL), Python (Programming Language), Release Management, Power BI, Standard Sql, Runbook, SQL Databases, SQL Server Integration Services, Workflow Management Systems, Data Logging, File Transfer Protocol (FTP), Apache Spark, Technical Debt, AWS CodeDeploy, Data Lakes, Pyspark, Integration Tests, Comarch, Deployment Automation, Machine Learning Operations, Software Version Control, Data Pipelines, Databricks
- **Published:** October 31, 2026
- **Apply:** https://www.reed.co.uk/jobs/cdi-data-engineer/57509292

## About the Role

* Strong SQL, Python/Spark and Databricks pipeline engineering background.
 * Large-scale transaction, payment, loyalty, EPOS/WRE and identity-matching work history
 * Historical/full/incremental loads, high-water marks and reconciliation.
 * Azure DevOps PRs, unit testing, CI/CD and production release support.

Nice to have: Reporting/Power BI dataset exposure desirable. SFMC/CIS integration and multi-market SCI knowledge desirable., * Strong hands-on background with Databricks, SQL, Python/PySpark ETL/ELT Data-pipeline migration

 * Legacy-to-cloud modernisation
 * Production troubleshooting
 * Dependency analysis
 * Pipeline repointing
 * Error handling and recovery
 * Data validation/reconciliation
 * Integration testing
 * Production support / L2-L3 troubleshooting

Nice to have: AWS-to-Databricks migration Informatica / SSIS / legacy ETL SFMC Comarch SFTP/file interfaces Azure/AWS CI/CD

## Description

* Implement and maintain SCI base, fact and curated pipelines.
 * Deliver Elavon/PAR/WRE/payment and loyalty matching.
 * Support ICE-to-EV data engineering and dashboard datasets.
 * Implement DO-site and market-specific logic.
 * Investigate volume drops, match-rate changes and data completeness defects.
 * Execute controlled reloads, production validation, DQ/freshness controls and BAU support.
 * Prepare PRs, tests, technical designs, CAB/release evidence and stakeholder

Role 2: BNA Production-Readiness, MLOps & Integration Engineer:

 * Automate end-to-end BNA orchestration
 * Reliable retry/recovery and idempotent processing
 * Production monitoring and alerting automated activation and acknowledgement handling
 * Reducing manual intervention Stable QA production deployment process
 * Runbooks and operational handover
 * Reducing runtime and production failures

Role 3: CIS/SCI Integration & Post-Migration Stabilisation Engineer:

 * Repointing remaining CIS/SCI dependencies
 * Stabilising migrated pipelines
 * Reusable error-handling patterns
 * Reducing legacy technical debt
 * Removing dual-platform monitoring
 * Resolving SFMC/Comarch integration issues
 * Reducing production support burden
 * Clear handover into BAU, * Strong hands-on background in Databricks, PySpark, Python and SQL Production pipeline engineering and troubleshooting
 * Workflow orchestration and dependency management Idempotency, retry/restart and failure-recovery patterns CI/CD, version control and automated deployment
 * Worked across monitoring, logging and operational alerting
 * Completed data-quality and reconciliation controls API/file/SFTP integration patterns

Nice to have: Delta Lake / Delta MERGE Databricks Asset Bundles Azure DevOps MLOps/model deployment work history SFMC or Comarch integration Customer decisioning / personalisation platforms