> Markdown version of [/jobs/ext/2263984-senior-data-engineer-databricks](https://www.wearedevelopers.com/jobs/ext/2263984-senior-data-engineer-databricks). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer (Databricks) - **Company:** EXL - **Location:** London, UK - **Experience:** Expert - **Salary:** £63,874.0 - **Contract:** Permanent contract - **Skills:** Amazon Web Services, JIRA, Microsoft Azure, Big Data, Information Engineering, Data Governance, Extract Transform Load (ETL), Data Transformation, Data Systems, Data Vault Modeling, Data Warehousing, DevOps, Distributed Systems, Meta-Data Management, SQL Azure, Azure Data Lake, SQL Databases, Enterprise Data Management, Azure Data Factory, Apache Spark, Multi-Cloud, Git, Data Layers, Pyspark, Azure Synapse Analytics, Software Version Control, Data Pipelines, Databricks - **Published:** August 27, 2026 - **Apply:** https://www.adzuna.co.uk/jobs/details/5856975762 ## About the Role * Insurance industry experience is essential, with a strong understanding of policy, claims, and regulatory data. * Minimum of 5 years' management experience required - leading technical delivery or overseeing project teams within data engineering or integration environments. * 8-10 years' experience in data engineering and integration, ideally in enterprise-scale environments. * Deep expertise in Azure Databricks, Azure Data Factory, Azure Synapse, and Azure Data Lake. * Hands-on experience with Spark (PySpark/SparkSQL), ETL/ELT design, and data pipeline orchestration. * Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts. * Experience working with large, complex datasets and distributed computing environments. * Proficiency in SQL, version control (e.g. Git), and CI/CD pipelines. * Familiarity with Agile delivery and DevOps practices (Azure DevOps, JIRA, etc.). Preferred Skills & Experience: * Knowledge of EDM (Enterprise Data Management) practices, data governance, metadata management, and lineage tools. * Exposure to multi-cloud environments (AWS or GCP). * Understanding of modern data architecture philosophies (Dimensional, ODS, Data Vault) and real-time cloud data capabilities. ## Description You'll work across data engineering, modelling, and transformation initiatives - building scalable pipelines, optimising performance, and ensuring best practices in data quality and governance. This is a hands-on technical role requiring strong Azure and Databricks experience, coupled with solid Insurance domain knowledge. As part of this role, you'll have the opportunity to work on high-impact Insurance data transformation programmes, collaborating with experienced data professionals in a forward-thinking, innovative environment. You'll gain exposure to cutting-edge Azure and Databricks technologies. As part of your duties, you will be responsible for: * Design, build, and deploy scalable data pipelines using Azure Databricks, Azure Data Factory, and Azure SQL Database. * Optimise Databricks environments for performance, cost, and security. * Develop and maintain data integration, ETL/ELT processes, and data quality frameworks. * Translate business requirements into data models, ensuring consistency and accuracy across data layers. * Work with stakeholders to deliver effective data solutions, analytics. * Create technical documentation including data flow diagrams, architecture design, and test plans. * Stay current with emerging data engineering and Azure ecosystem technologies to drive continuous improvement. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [Improving quality with Agentic AI with Rovo Dev and Xray](https://www.wearedevelopers.com/videos/2005-improving-quality-with-agentic-ai-with-rovo-dev-and-xray) - [From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform](https://www.wearedevelopers.com/videos/100018-from-devops-to-scaled-devops-how-we-re-rebuilding-continuous-delivery-as-a-platform) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Git for Code Reviews](https://www.wearedevelopers.com/videos/429-git-for-code-reviews) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Top-Paying Tech Jobs (with Salaries)](https://www.wearedevelopers.com/magazine/372-top-paying-tech-jobs-with-salaries)