Lead Data Engineer
Role details
Job location
Tech stack
Job description
The Lead Data Engineer will provide hands-on technical leadership in Azure cloud and Databricks-based solutions within our Enterprise Data Platform. The role requires strong expertise in Azure cloud services, Databricks, data engineering, and DevOps, leading a cross-functional team to build, deploy, and support high-performance data-driven solutions. The role involves:
- Interpreting Outcomes and user stories and translating them into technical solutions.
- Creating innovative solution designs for domain and enterprise data products.
- Overseeing Data Analysts to support detailed data discovery.
- Overseeing data modelling for Finance and Enterprise data products.
- Leading Product Increment planning to break down solutions into Features and Epics for incremental delivery.
- Designing and implementing scalable data solutions on Azure and Databricks within the assigned domain.
- Ensuring appropriate engineering standards are applied to maintain data quality, performance and reliability.
Duties/Responsibilities
- Work with Product Owners and Business Analysts to understand Outcomes, refine user stories.
- Lead solution design for Finance and Enterprise data products, ensuring alignment with enterprise patterns and guardrails.
- Direct and collaborate with Data Analysts on detailed data discovery, source understanding and requirements refinement.
- Oversee logical and physical data modelling for Finance and Enterprise data products, working closely with architecture where required.
- Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake).
- Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established patterns.
- Apply security, access control and compliance standards for Azure and Databricks in collaboration with platform and security teams.
- Support monitoring, logging and basic cost optimisation for the team's data products.
- Support the development of DevOps practices within the team, including reducing technical debt and improving automation over time., Role: Lead Data Engineer - Azure/Databricks Location: Sheffield Working Model: Hybrid - 2 days per week in person Salary: Up to £85k depending on experience Own the data platform. Shape the future architecture. This role is for a hands-on data engineering leader who...
Requirements
Lead Data Engineers are expected to have strong capability in at least three of the following areas of engineering practice. Core skills
- Automation including testing of data pipelines and data products.
- Strong teamwork, communication and problem-solving skills to collaborate effectively with cross-functional teams.
- Awareness of security principles and best practices to ensure secure data solutions.
- Commitment to continuous learning and staying current with Azure, Databricks and data engineering trends.
- Strong experience working within an agile development methodology, ideally Scaled Agile (SAFe or similar).
- Excellent time and self-management through effective planning and prioritisation of tasks.
- Proven and demonstrable data engineering capability.
- Ability to influence within the team and communicate clearly with technical and non-technical stakeholders.
Data Engineer (New Technology / Microsoft)
- Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous).
- Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous).
- Experience contributing to CI/CD pipelines (Azure DevOps, GitHub Actions, Terraform).
- Scripting and programming skills (Python advantageous).
- Good understanding of DevOps and automation concepts (e.g. YAML pipelines, IaC).
- Solid understanding of cloud security, compliance and governance principles.
- Experience working with Databricks and Azure in a product or Scaled Agile delivery environment.
Qualifications
- Degree level IT or technical/scientific subject (or equivalent experience).
- Microsoft Azure Data Engineer or Solutions Architect certification (desirable).
- Databricks Certified Data Engineer or Machine Learning Associate (desirable).
- Experience with streaming solutions (Kafka, Event Hubs, Spark Streaming) (desirable).
- Knowledge of machine learning and AI on Databricks (desirable)., We are seeking a Lead Data Engineering Consultant with proven experience in leading and developing data engineering platforms. The ideal candidate will possess hands-on expertise in the following areas: Apply below after reading through all the details and supporting..., Job Description - Gloucester location - hybrid working when possible - Must hold active Enhanced DV Clearance (West) - Competitive Salary DOE - 6% bonus, 25 days holiday, clearance bonus - Experience in Data Pipelines, ETL processing, Data Integration, Apache, SQL/NoSQL,..., Lead Data Engineer - Databricks - £75-80k + bonus + benefits - Basingstoke 3 days a week The Opportunity: We are looking for a Lead Data Engineer with good Databricks expeirence to join a Basingstoke based organisation who are investing heavily in their Digital...
Benefits & conditions
Benefits of working at Canada Life We believe in recognising and rewarding our people, so we offer a competitive salary and benefits package that's regularly reviewed. As a Canada Life UK colleague, you'll receive a competitive salary and comprehensive reward package including a generous pension and bonus scheme, along with, income protection, private medical insurance and life assurance. We have a fantastic number of other benefits and support services as well as regular personal and professional development. How we work at Canada Life Our culture is unique and incredibly important to us. We care about doing the right thing for our people, customers and community and helping others to build better futures. Our blueprint behaviours shape and influence how we work, and are central to the relationships we have with others. Every day we are encouraged to be more curious, own the outcome, face into things together and find a way forward. We want colleagues to have rewarding careers with us so we invest in the development of our people, technology and workplaces. That's why we offer a range of training, flexible working and opportunities to grow and develop. Diversity and inclusion Building an inclusive workplace with a diverse workforce where everyone can feel they belong and achieve their potential regardless of gender, ethnicity or any other characteristic is a key commitment for us. We are proud of the progress we're making in DEI, and we continue for it to be a significant focus. "At Canada Life we believe in the power of great people from different backgrounds, experiences and perspectives coming together to build better futures. Emerging talent is crucial to our growth and creating an environment that continues to inspire us all." Nick Harding, Chief People Officer, Canada Life UK We appreciate that everyone has different work and life responsibilities. We're happy to discuss flexible working arrangements, including part time, for any of our roles should this be a requirement for you., Job Description - Senior Engineer/Architect - Leading Technology AI Brand - SaaS - Platform based Technology Services - London/City - £70-100k salary + equity package Our client a global technology leader is currently looking for a Senior/Lead Data Engineer to work with...
BI and Data Engineering Lead
The Wilf Ward Family Trust £40,000
BI and Data Engineering Lead A variety of soft skills and experience may be required for the following role Please ensure you check the overview below carefully. Salary: £40,000£45,000 per annum Reporting to: Digital Delivery Manager Location: Trust sites (with travel...
Data Engineering Lead- Manchester (hybrid) - up to £80K DOE
Forward Role
Data Engineering Lead- Manchester (hybrid) - up to £80K DOE We are working with a fast-growing fintech business in Manchester who are looking for a Data Team Lead. They have new PE investment behind them and a major transformation programme underway. This is one of the..., Job Description Lead Data Platform Engineer £75,000 - £85,000 Remote You are a well-experienced Lead Data Platform Engineer, looking to take ownership of a modern lakehouse platform and shape best-in-class data engineering practices. THE COMPANY This industry-leading... © 2026, Jobsora.com
