> Markdown version of [/jobs/ext/1327026-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/1327026-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Data Engineer

- **Company:** Pioneer Search Ltd
- **Location:** London, UK
- **Experience:** Expert
- **Salary:** £95,000.0 - £110,000.0
- **Contract:** Permanent contract
- **Skills:** Agile Methodology, Microsoft Azure, Big Data, Code Review, Continuous Integration, Information Engineering, DevOps, Distributed Computing Environment, Python (Programming Language), Cloud Services, Freeform SQL, Pyspark, Data Management, Data Pipelines, Serverless Computing, Service Stack, Databricks
- **Published:** July 18, 2026
- **Apply:** https://www.adzuna.co.uk/jobs/details/5805146827

## About the Role

* Strong commercial experience building data engineering solutions using Python, PySpark and Databricks.
 * Proven experience designing and optimising complex SQL queries and working with large-scale datasets.
 * Strong experience across the Azure technology stack, including Azure Functions and cloud-native data services.
 * Previous experience leading or mentoring Data Engineers within an Agile environment.
 * Strong understanding of distributed data processing, modern data engineering practices and CI/CD.
 * Experience building scalable, reliable and high-performing data platforms.
 * Experience within Financial Services or Insurance is highly advantageous.

## Description

As a Lead Data Engineer, you'll play a key role in designing and evolving a modern cloud-based data platform built on Databricks and Azure. You'll lead the development of scalable data pipelines using Python and PySpark, enabling high-quality data to power underwriting, analytics and business-critical decision-making across the organisation.

Working closely with Product Owners, Architects and business stakeholders, you'll provide technical leadership throughout the full data lifecycle. Alongside remaining hands-on, you'll mentor a team of Data Engineers, champion engineering best practices and help shape the future of the organisation's cloud data platform., * Lead and mentor a team of Data Engineers, supporting the successful delivery of data engineering projects from design through to production.

 * Design, develop and optimise scalable data pipelines using Python, PySpark and Databricks.
 * Build and maintain cloud-native data solutions across the Azure ecosystem, including Azure Functions and related services.
 * Collaborate with Product Owners, Architects and business stakeholders to define technical solutions and deliver against business priorities.
 * Drive engineering best practices through code reviews, mentoring and continuous improvement initiatives.
 * Identify opportunities to improve platform performance, data quality, scalability and operational efficiency.
 * Champion automation, testing, CI/CD and modern engineering practices across the data engineering function.


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology)
- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform](https://www.wearedevelopers.com/videos/100018-from-devops-to-scaled-devops-how-we-re-rebuilding-continuous-delivery-as-a-platform)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)

## Related Articles

- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)
- [Fully Remote Software Engineer Jobs](https://www.wearedevelopers.com/magazine/447-fully-remote-software-engineer-jobs)