> Markdown version of [/jobs/ext/3721664-senior-data-engineer-databricks](https://www.wearedevelopers.com/jobs/ext/3721664-senior-data-engineer-databricks). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Senior Data Engineer (Databricks)

- **Company:** EXL Service
- **Location:** London, UK
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** Amazon Web Services, JIRA, Microsoft Azure, Big Data, Information Engineering, Data Governance, Extract Transform Load (ETL), Data Transformation, Data Systems, Data Vault Modeling, Data Warehousing, DevOps, Distributed Systems, Meta-Data Management, SQL Azure, Azure Data Lake, SQL Databases, Enterprise Data Management, Azure Data Factory, Apache Spark, Multi-Cloud, Git, Data Layers, Pyspark, Azure Synapse Analytics, Software Version Control, Data Pipelines, Databricks
- **Published:** October 14, 2026
- **Apply:** https://www.totaljobs.com/job/senior-data-engineer-databricks/exl-service-job108160628

## About the Role

* Insurance industry experience is essential, with a strong understanding of policy, claims, and regulatory data.
 * Minimum of 5 years' management experience required - leading technical delivery or overseeing project teams within data engineering or integration environments.
 * 8-10 years' experience in data engineering and integration, ideally in enterprise-scale environments.
 * Deep expertise in Azure Databricks, Azure Data Factory, Azure Synapse, and Azure Data Lake.
 * Hands-on experience with Spark (PySpark/SparkSQL), ETL/ELT design, and data pipeline orchestration.
 * Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts.
 * Experience working with large, complex datasets and distributed computing environments.
 * Proficiency in SQL, version control (e.g. Git), and CI/CD pipelines.
 * Familiarity with Agile delivery and DevOps practices (Azure DevOps, JIRA, etc.).

Preferred Skills & Experience:

 * Knowledge of EDM (Enterprise Data Management) practices, data governance, metadata management, and lineage tools.
 * Exposure to multi-cloud environments (AWS or GCP).
 * Understanding of modern data architecture philosophies (Dimensional, ODS, Data Vault) and real-time cloud data capabilities.

## Description

You'll work across data engineering, modelling, and transformation initiatives - building scalable pipelines, optimising performance, and ensuring best practices in data quality and governance. This is a hands-on technical role requiring strong Azure and Databricks experience, coupled with solid Insurance domain knowledge.

As part of this role, you'll have the opportunity to work on high-impact Insurance data transformation programmes, collaborating with experienced data professionals in a forward-thinking, innovative environment. You'll gain exposure to cutting-edge Azure and Databricks technologies.

As part of your duties, you will be responsible for:

 * Design, build, and deploy scalable data pipelines using Azure Databricks, Azure Data Factory, and Azure SQL Database.
 * Optimise Databricks environments for performance, cost, and security.
 * Develop and maintain data integration, ETL/ELT processes, and data quality frameworks.
 * Translate business requirements into data models, ensuring consistency and accuracy across data layers.
 * Work with stakeholders to deliver effective data solutions, analytics.
 * Create technical documentation including data flow diagrams, architecture design, and test plans.
 * Stay current with emerging data engineering and Azure ecosystem technologies to drive continuous improvement.