Lead Data Engineer

Technical Lead Databricks Engineer required for a greenfield Azure data platform build. There is no legacy estate and no migration. The end client is a newly formed organisation whose service is expected to operate at national scale following incoming legislation, so the platform must be engineered for very significant growth.

Discovery is largely complete. The platform decision has been made: Azure with Databricks, and Microsoft Purview for governance. You will turn the target architecture into a working platform, setting engineering standards, leading the build and staying hands on throughout.

This is a client facing role. We are looking for someone with a track record of delivering data solutions directly to clients, usually from a consultancy or systems integrator background, who can work with the client team day to day, explain technical decisions clearly and help them build their own capability.

Responsibilities

• Lead the engineering build of the Databricks Lakehouse on Azure, working to the target architecture set by the Data Architect
• Act as the day to day technical contact for the client team, running working sessions, demos and technical walkthroughs
• Translate architectural decisions into clear delivery plans and explain trade-offs to technical and non-technical stakeholders
• Build and review ingestion and transformation pipelines across medallion layers, for both batch and streaming
• Implement Unity Catalog structure, access controls and lineage, and integrate with Microsoft Purview
• Set engineering standards for code quality, testing, data quality controls, CI/CD and environment promotion
• Automate deployment with Databricks Asset Bundles, Terraform or Bicep, and Azure DevOps or GitHub Actions
• Tune performance and cost, including Spark optimisation and cluster and warehouse design
• Mentor engineers within the client team, including code reviews and knowledge transfer
• Work alongside the Data Architect, enterprise architects, integrations architect and data governance specialist

• Proven experience delivering data solutions directly to clients, ideally in a consultancy or systems integrator environment
• Strong client facing skills, with confidence presenting, gathering requirements and handling stakeholder questions and challenge
• Technical lead or lead engineer experience on enterprise scale Databricks platforms on Azure
• Strong hands on data engineering background with Python or PySpark and advanced SQL
• Delta Lake, Unity Catalog, medallion architecture, Databricks Workflows and Auto Loader
• Declarative pipelines (Delta Live Tables or Lakeflow) and Structured Streaming
• CI/CD for Databricks using Databricks Asset Bundles, Azure DevOps or GitHub Actions, and Terraform or Bicep
• Azure Data Factory, Azure SQL, Data Lake Storage Gen2
• Entra ID, Key Vault, private endpoints, RBAC
• Experience of data governance and lineage, ideally with Microsoft Purview
• UK GDPR and DPA 2018 in practice, including public personal data

Desirable

• Greenfield platform build experience
• Experience mentoring or upskilling client engineering teams
• Databricks certification, for example Data Engineer Professional
• Event driven design and high volume ingestion
• Dimensional and data vault modelling
• Regulated or legislation driven programme experience