Lead Azure Databricks Engineer
Role details
Job location
Tech stack
Job description
We are seeking a Lead Azure Databricks Engineer to join a specialist consultancy focused on transforming enterprises through modern cloud data platforms, advanced analytics and next-generation engineering capabilities.
This role requires strong experience working within the Lloyd's of London or wider London Market insurance environment, with the ability to deliver data solutions that meet strict regulatory, audit and operational standards.
You will take ownership of designing, building and optimising large-scale Azure data solutions, working closely with business and technical stakeholders to translate complex data challenges into secure, scalable and production-ready platforms. You will help shape cloud data strategy, define engineering best practice and guide delivery across cross-functional teams.
Responsibilities
- Build and optimise Azure data platforms using ADF, Data Lake, Azure Functions and Databricks
- Deliver reliable, high-quality data pipelines
- Work with stakeholders to ensure solutions meet Lloyd's of London regulatory, audit and data governance requirements
- Work with stakeholders to make sure solutions meet regulatory requirements.
- Promote best practices in engineering, automation and cloud design.
- Lead and mentor engineers, setting technical standards.
- Improve platform performance, resilience and cost efficiency.
Requirements
- Extensive engineering experience across Azure Data Services and Databricks in enterprise environments.
- Proven delivery experience within the Lloyd's of London or London Market insurance sector
- Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns and data modelling.
- Strong Python, PySpark and Spark SQL capability
- Solid understanding of data governance
- Proven track record of ownership, delivery and operating effectively in fast-paced environments.
Benefits & conditions
- Salary of up to £95,000
- Performance related bonus scheme
- Hybrid working - Up to 3 days working from home
- Comprehensive benefits package
