Lead Data Engineer

As a Lead Data Engineer, you’ll play a key role in designing and evolving a modern cloud-based data platform built on Databricks and Azure. You’ll lead the development of scalable data pipelines using Python and PySpark, enabling high-quality data to power underwriting, analytics and business-critical decision-making across the organisation.

Working closely with Product Owners, Architects and business stakeholders, you’ll provide technical leadership throughout the full data lifecycle. Alongside remaining hands-on, you’ll mentor a team of Data Engineers, champion engineering best practices and help shape the future of the organisation’s cloud data platform., * Lead and mentor a team of Data Engineers, supporting the successful delivery of data engineering projects from design through to production.

• Design, develop and optimise scalable data pipelines using Python, PySpark and Databricks.
• Build and maintain cloud-native data solutions across the Azure ecosystem, including Azure Functions and related services.
• Collaborate with Product Owners, Architects and business stakeholders to define technical solutions and deliver against business priorities.
• Drive engineering best practices through code reviews, mentoring and continuous improvement initiatives.
• Identify opportunities to improve platform performance, data quality, scalability and operational efficiency.
• Champion automation, testing, CI/CD and modern engineering practices across the data engineering function.

• Strong commercial experience building data engineering solutions using Python, PySpark and Databricks.
• Proven experience designing and optimising complex SQL queries and working with large-scale datasets.
• Strong experience across the Azure technology stack, including Azure Functions and cloud-native data services.
• Previous experience leading or mentoring Data Engineers within an Agile environment.
• Strong understanding of distributed data processing, modern data engineering practices and CI/CD.
• Experience building scalable, reliable and high-performing data platforms.
• Experience within Financial Services or Insurance is highly advantageous.

• Opportunity to shape the future of a modern, cloud-native data platform built on Databricks and Azure.
• Hands-on leadership role combining technical delivery with engineering strategy.
• Work with modern technologies including Python, PySpark, Databricks and the wider Azure ecosystem.
• Collaborative engineering culture focused on innovation, quality and continuous improvement.
• Competitive salary, bonus and comprehensive benefits package.
• Hybrid working from a modern London office.

Our client is a leading specialist insurance provider, recognised for its innovative approach to technology and data. Operating globally, they are investing heavily in their modern data platform, leveraging cloud technologies to deliver scalable, high-quality data solutions that support underwriting, analytics and business decision-making.