> Markdown version of [/jobs/ext/3026760-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/3026760-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Data Engineer

- **Company:** Stott and May
- **Location:** UK (Remote available)
- **Experience:** Expert
- **Contract:** Temporary contract
- **Skills:** Artificial Intelligence, Microsoft Azure, Code Review, Data Architecture, Information Engineering, Data Governance, Extract Transform Load (ETL), Python (Programming Language), Standard Sql, SQL Databases, Data Lakes, Pyspark, Data Pipelines, Databricks
- **Published:** September 22, 2026
- **Apply:** https://find.stottandmay.com/job/lead-data-engineer-79348/apply

## About the Role

* 7+ years' experience in Data Engineering.
 * 3+ years' hands-on experience delivering large-scale Databricks implementations within financial services.
 * Strong experience within Insurance, Banking, Asset Management or FinTech.
 * Advanced Python, PySpark and SQL skills.
 * Deep knowledge of Databricks Lakehouse architecture and Delta Lake.
 * Experience with at least one major cloud platform: AWS, Azure or GCP.
 * Proven experience leading or mentoring Data Engineering teams.
 * Strong stakeholder management and communication skills.
 * Experience working within regulated environments, with an understanding of data governance and compliance requirements.

## Description

We are looking for an experienced Lead Data Engineer to join a leading insurance client, providing technical leadership across data engineering initiatives and supporting the development of a high-performing engineering team.

The role is both hands-on and leadership-focused, with Databricks at the centre of the technology stack alongside Python, PySpark and SQL. You will take ownership of the Databricks platform and architecture while working closely with senior stakeholders across Risk, Quantitative Finance, Compliance and Data Science., * Own the Databricks architecture, strategy and roadmap, ensuring performance, availability and cost efficiency.

 * Design, build and operationalise scalable ETL/ELT data pipelines using Databricks, Python and SQL.
 * Develop and optimise solutions using Databricks Lakehouse and Delta Lake.
 * Implement data governance, lineage, controls and audit frameworks within a regulated environment.
 * Lead, mentor and upskill a team of Data Engineers, establishing technical standards and conducting code reviews.
 * Provide technical direction across data engineering projects and drive engineering best practices.
 * Work closely with Risk, Quantitative Finance, Compliance and Data Science teams to translate business requirements into technical solutions.
 * Communicate complex technical concepts and architecture clearly to senior and executive stakeholders.


## Related Videos

- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)
- [Fully Orchestrating Databricks from Airflow](https://www.wearedevelopers.com/videos/336-fully-orchestrating-databricks-from-airflow)
- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [OLTP in the Lakehouse: Redefining Data for AI Workloads](https://www.wearedevelopers.com/videos/2038-oltp-in-the-lakehouse-redefining-data-for-ai-workloads)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)

## Related Articles

- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk)
- [Fullstack Developer Salary UK](https://www.wearedevelopers.com/magazine/251-fullstack-developer-salary-uk)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)