Lead Data Engineer
Role details
Job location
Tech stack
Job description
ARCA Resourcing is proud to be partnered with an innovative and leading retailer that is investing heavily in its data capabilities to drive smarter decision-making across the business. As part of this transformation, we're looking for a Lead Data Engineer to play a key role in shaping and delivering the organisation's modern Enterprise Data Platform.
This is an opportunity to combine hands-on engineering with technical leadership, working with cutting-edge technologies while mentoring engineers and influencing the direction of a high-impact data platform.
The Opportunity
The organisation operates a modern data stack built around a Databricks-based lakehouse architecture, alongside a Customer Data Platform (CDP), enterprise analytics tooling, and self-service reporting capabilities.
As a Lead Data Engineer, you will oversee multiple delivery squads, ensuring consistent engineering standards and guiding the design and delivery of scalable data solutions. You'll coordinate technical delivery across teams while remaining hands-on with architecture, pipelines, and best practice development.
This role is ideal for someone who enjoys leading teams, shaping engineering standards, and building high-performance data platforms at scale.
Key Responsibilities
- Provide technical leadership across multiple delivery squads, ensuring consistent data engineering standards and patterns.
- Coordinate technical solutions and align team efforts across initiatives.
- Translate functional designs into scalable and reusable data engineering solutions.
- Collaborate closely with solution architects, business analysts, and stakeholders to refine requirements and technical approaches.
- Design and implement high-performance ELT pipeline patterns using Databricks.
- Mentor and develop engineers across the team, creating learning and development plans.
- Champion engineering best practices, including code quality, testing, and documentation.
- Conduct peer reviews to ensure high-quality and reliable deliverables.
- Optionally contribute to on-call platform support where required.
Requirements
Technical Expertise
- Strong experience with Databricks, Apache Spark, and cloud-based data architectures (preferably Azure).
- Excellent programming capability in Python and SQL.
- Experience orchestrating pipelines with Azure Data Factory and Databricks Workflows.
- Solid understanding of Lakehouse architectures and modern data warehousing principles.
- Experience with data modelling, ELT/ETL pipelines, and testing frameworks.
- Familiarity with CI/CD pipelines, version control, and DevOps practices.
- Knowledge of security, governance, and cost optimisation in cloud data platforms.
Leadership Experience
- Proven experience leading engineering teams or coordinating multiple squads.
- Experience aligning engineering work with strategic project objectives.
- Ability to mentor and develop engineers, driving technical excellence across teams.
- Strong problem-solving, debugging, and architectural thinking skills.
Desirable
- Degree in IT, Computer Science, or related discipline. xhkmmrq
Benefits & conditions
Job Description Are you the right applicant for this opportunity Find out by reading through the role overview below. Data Engineer (Permanent)Salary: Up to £60,000Location: London Victoria (1 day per week onsite, rest remote)The Role We are seeking a skilled Data...
