Lead Data Engineer

• Lead the design, development and optimisation of a Databricks based data platform, ensuring reliability, performance and scalability
• Design enterprise scale data architectures on Databricks and cloud platforms such as Azure, AWS or GCP
• Define lakehouse standards, data modelling approaches and governance frameworks
• Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies
• Establish security, access control, data quality and compliance standards
• Optimise performance, scalability and cloud cost management
• Lead and mentor a small team of data engineers, setting technical standards and reviewing code
• Partner with data science, analytics and product teams to ensure the platform supports their work
• Own data quality, governance and observability across pipelines
• Support AI and ML enablement including feature stores and model deployment patterns
• Translate business requirements into technical solutions
• Contribute to platform roadmap, tooling selection and continuous improvement

• Strong experience as a senior or lead data engineer building and scaling production data platforms
• Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows
• Python and SQL proficiency
• Infrastructure as code experience using Terraform, Bicep or CloudFormation
• Cloud data platform experience, ideally Azure including Azure Data Factory, ADLS Gen2 and Azure DevOps
• Strong understanding of data modelling, ETL and ELT patterns, distributed processing
• Experience with security, governance and data platform operations
• Experience leading or mentoring engineers, with strong stakeholder management
• Rigorous approach to data quality and testing
• Databricks certification

Desirable

• Unity Catalog, MLflow and Databricks Asset Bundles
• dbt or similar transformation frameworks
• Retail or CPG data such as POS, inventory, supply chain
• Streaming technologies such as Kafka or Event Hubs

Education

• Degree in computer science, data engineering or related field, or equivalent practical experience