Lead Data Engineer

Stott and May

10 days ago

Apply on find.stottandmay.com

Prepare application

Role details

Contract type

Temporary contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

3 years minimum

Working hours

Regular working hours

Job source

find.stottandmay.com

Tech stack

Artificial Intelligence Microsoft Azure Code Review Data Architecture Information Engineering Data Governance Extract Transform Load (ETL) Python (Programming Language) Standard Sql SQL Databases Data Lakes Pyspark

+2 more

Data Pipelines Databricks

Job description

We are looking for an experienced Lead Data Engineer to join a leading insurance client, providing technical leadership across data engineering initiatives and supporting the development of a high-performing engineering team.

The role is both hands-on and leadership-focused, with Databricks at the centre of the technology stack alongside Python, PySpark and SQL. You will take ownership of the Databricks platform and architecture while working closely with senior stakeholders across Risk, Quantitative Finance, Compliance and Data Science., * Own the Databricks architecture, strategy and roadmap, ensuring performance, availability and cost efficiency.

Design, build and operationalise scalable ETL/ELT data pipelines using Databricks, Python and SQL.
Develop and optimise solutions using Databricks Lakehouse and Delta Lake.
Implement data governance, lineage, controls and audit frameworks within a regulated environment.
Lead, mentor and upskill a team of Data Engineers, establishing technical standards and conducting code reviews.
Provide technical direction across data engineering projects and drive engineering best practices.
Work closely with Risk, Quantitative Finance, Compliance and Data Science teams to translate business requirements into technical solutions.
Communicate complex technical concepts and architecture clearly to senior and executive stakeholders.

Requirements

7+ years’ experience in Data Engineering.
3+ years’ hands-on experience delivering large-scale Databricks implementations within financial services.
Strong experience within Insurance, Banking, Asset Management or FinTech.
Advanced Python, PySpark and SQL skills.
Deep knowledge of Databricks Lakehouse architecture and Delta Lake.
Experience with at least one major cloud platform: AWS, Azure or GCP.
Proven experience leading or mentoring Data Engineering teams.
Strong stakeholder management and communication skills.
Experience working within regulated environments, with an understanding of data governance and compliance requirements.