> Markdown version of [/jobs/ext/2019692-senior-data-engineer](https://www.wearedevelopers.com/jobs/ext/2019692-senior-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer - **Company:** CMSPI - **Location:** Manchester, UK - **Experience:** Expert - **Salary:** £81,967.0 - **Contract:** Permanent contract - **Skills:** Application Programming Interfaces (APIs), Systems Engineering, Automation of Tests, Microsoft Azure, Code Review, Continuous Integration, Information Engineering, Data Integration, Extract Transform Load (ETL), Data Retrieval, Data Security, DevOps, File Transfer, File Transfer Protocol (FTP) SSL Extension, Python (Programming Language), Operational Databases, Role-Based Access Control, Standard Sql, Secure Coding, SQL Databases, Workflow Management Systems, File Transfer Protocol (FTP), Apache Spark, Infrastructure Automation Frameworks, Information Technology, Data Management, Software Version Control, Data Pipelines, Databricks - **Published:** August 11, 2026 - **Apply:** https://www.adzuna.co.uk/jobs/details/5836613307 ## About the Role · 3+ years professional experience as a Data Engineer, DevOps Engineer or Systems Engineer delivering production data platforms. · Strong hands-on experience with Azure Databricks, including Spark/SQL, workflow orchestration and production lakehouse data engineering. · Advanced proficiency in Python for data retrieval, processing, automation, integration and reusable engineering libraries. · Experience designing and operating ETL/ELT pipelines with data quality controls, schema handling, monitoring and failure recovery. · Expertise in CI/CD, version-controlled deployment and automated testing for data or platform workloads. · Experience integrating data through APIs and secure file transfer (SFTP/FTPS), including secure handling of credentials and secrets. · Strong Microsoft Azure knowledge, including ADLS, Azure DevOps and Entra/RBAC, with experience designing and deploying infrastructure. · Strong production troubleshooting, problem-solving and operational support skills. · Degree in Computer Science, Mathematics, Physics or other STEM related subject. ## Description As a Senior Data Engineer, you will be an integral part of a cross-functional feature development squad You will play a key role in the evolution of our payments intelligence platform towards a Databricks-native and increasingly agentic operating model. You will design, build and operate secure data ingestion, orchestration and platform capabilities, using Databricks as the first-choice execution platform and Azure services where appropriate. The focus remains on production data engineering rather than AI model development: creating reliable, observable and well-governed foundations that can be operated by both engineers and automated/agentic workflows. You will work across client onboarding, data retrieval, ELT, CI/CD and platform reliability to deliver continuous business value. JOB ROLEKey accountability of this role: Delivery, support and continuous improvement of secure, scalable, Azure/Databricks data capabilities and infrastructure on our payments intelligence platform. Key responsibilities will include: · Collaborate within a cross-functional squad to define and deliver scalable, secure data and platform capabilities aligned with business outcomes. · Design Databricks-native solutions as the default, using Spark/SQL, Lakeflow Jobs, Unity Catalog, Volumes and Databricks Asset Bundles, with Azure services used where they add value. · Build and operate data retrieval and ingestion across APIs, SFTP/FTPS and browser automation, migrating suitable orchestration from external services into Databricks-native workflows. · Own client data onboarding flows, including configuration, historical backfills, validation, data quality, monitoring and operational handover. · Build reusable platform capabilities and guardrails that enable agentic workflows to safely automate onboarding, operational diagnosis and routine engineering tasks, with human approval where required. · Engineer for production reliability through observability, alerting, retries/idempotency, failure recovery, performance and cost optimisation; investigate complex failures and drive root-cause fixes. · Deliver changes through version control, automated testing, CI/CD and Infrastructure as Code, maintaining secure coding and deployment standards through code review. · Apply strong security and governance practices across Unity Catalog, Entra/RBAC, secrets, credentials and data access. · Use advanced Python with strong SQL/Spark skills to develop maintainable, reusable libraries, frameworks and data pipelines. · Mentor data engineers and drive continuous improvement, evaluating new Databricks and automation capabilities against clear technical and business value. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Are Code Reviews Worth It? Insights from 16 Years of Review Data](https://www.wearedevelopers.com/videos/1135-are-code-reviews-worth-it-insights-from-16-years-of-review-data) - [Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source](https://www.wearedevelopers.com/videos/100151-cutting-llm-costs-without-cutting-quality-how-to-beat-proprietary-llms-with-fine-tuned-open-source) - [From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform](https://www.wearedevelopers.com/videos/100018-from-devops-to-scaled-devops-how-we-re-rebuilding-continuous-delivery-as-a-platform) - [Build a CI/CD pipeline to automate code reviews and ensure code quality](https://www.wearedevelopers.com/videos/349-build-a-ci-cd-pipeline-to-automate-code-reviews-and-ensure-code-quality) - [OLTP in the Lakehouse: Redefining Data for AI Workloads](https://www.wearedevelopers.com/videos/2038-oltp-in-the-lakehouse-redefining-data-for-ai-workloads) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Top-Paying Tech Jobs (with Salaries)](https://www.wearedevelopers.com/magazine/372-top-paying-tech-jobs-with-salaries) - [The Most Popular IT Jobs on the Market](https://www.wearedevelopers.com/magazine/376-the-most-popular-it-jobs-on-the-market) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london) - [The 12 Best Jobs for Software Engineers](https://www.wearedevelopers.com/magazine/401-the-12-best-jobs-for-software-engineers)