> Markdown version of [/jobs/ext/4763520-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/4763520-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Data Engineer - **Company:** The Money and Pensions Service - **Location:** Bedford, United Kingdom (Remote available) - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Microsoft Access, Application Programming Interfaces (APIs), Data Analysis, Application Frameworks, Automation of Tests, Microsoft Online Services, Code Review, Databases, Continuous Integration, Information Engineering, Data Infrastructure, Data Security, Data Systems, Identity and Access Management, Python (Programming Language), Metadata, Software Architecture, Release Management, DataOps, Data Logging, Data Processing, Azure Data Factory, Sql Optimization, Apache Spark, Technical Debt, Microsoft Fabric, Pyspark, Data Management, Azure Synapse Analytics, Software Version Control - **Published:** November 18, 2026 - **Apply:** https://www.jobs.service.gov.uk/jobs/6afdafa3ba6abc4abbf68bca/apply?applyMethod=WEBSITE ## About the Role * Strong experience designing, building and operating Microsoft cloud data platforms, including practical Microsoft Fabric experience and an understanding of Azure Synapse and Azure Data Factory in migration and integration scenarios. * Experience taking responsibility for data platform architecture and technical design, translating strategic objectives into implementation roadmaps and explaining decisions and trade-offs to technical and non-technical stakeholders. * Experience defining and embedding engineering standards, reusable frameworks and solution patterns across multiple engineers and delivery workstreams. * Experience managing, leading and mentoring Data Engineers, including setting objectives, supporting professional development, delegating responsibility and building a collaborative engineering culture. * Advanced SQL and strong Python/PySpark skills, with practical experience of Spark, lakehouse and warehouse architectures, layered data processing and analytical data modelling. * Experience delivering robust ingestion, integration and transformation pipelines using APIs, files and databases, including incremental processing, schema change, error handling and automated data quality validation. * Experience of platform migration or significant modernisation, including dependency assessment, reconciliation, testing and controlled transition to production. * Practical understanding of cloud identity and access management, data protection, governance, metadata, lineage, retention and secure data sharing, including how to implement these through engineering controls. * Experience of source control, code review, automated testing and CI/CD, with clear technical documentation and effective agile delivery practices. * Experience designing for production support, including observability, incident investigation, recovery and operational handover, and improving platform reliability, performance and cost. * Ability to work with business stakeholders, architects, governance, security and operational teams to clarify requirements, challenge assumptions and agree sustainable solutions. * Ability to evaluate emerging platform capabilities, assess risks and benefits, and make proportionate, evidence-based technology recommendations. * Ability to coordinate technical delivery across a larger team, balancing immediate delivery commitments with platform development, technical debt and longer-term goals. ## Description Join us at the Money and Pensions Service (MaPS) as a Lead Data Engineer. The Lead Data Engineer provides technical leadership for the architecture, development and operation of the MaPS data platform. Working with the Principal Data Engineer, you will translate strategic platform goals into practical architectural decisions, engineering approaches and a prioritised technical roadmap, enabling secure, reliable and reusable data products across MaPS. You will lead the technical evolution of the platform, including migration from Azure Synapse to Microsoft Fabric and support for expanding data needs. You will own data engineering technical standards and solution patterns, ensuring that platform development balances delivery needs with security, maintainability, performance and cost. As part of a larger data engineering team, you will manage, lead and mentor Data Engineers, building capability and shared ownership of engineering quality. This is a hands-on leadership role: you will guide technical delivery, resolve complex engineering problems and develop reusable platform capabilities rather than focusing solely on individual project solutions. You will work with architecture, governance, security, DataOps, analysts and delivery stakeholders to turn business requirements into sustainable data solutions. You will challenge constructively, explain technical choices and trade-offs clearly, and ensure that architectural decisions and engineering practices support MaPS services and the people who use them. The role * Lead data platform architecture and detailed technical design, working within agreed enterprise architecture and governance. * Translate strategic goals into implementation options, technical decisions, delivery sequencing and a platform development roadmap agreed with the Principal Data Engineer. * Own, document and maintain data engineering technical standards, reference architectures and reusable patterns, including data layering, modelling, naming, ingestion, transformation, testing and deployment. Embed their use through design reviews, code reviews and technical assurance. * Develop the platform's technology and shared capabilities. Evaluate new features through proportionate investigation and proofs of concept, recommending adoption based on business value, security, supportability, performance and cost rather than technology alone. * Lead technical delivery across multiple workstreams, resolving cross-project dependencies and ensuring solutions contribute to the shared platform rather than creating unnecessary duplication. * Line manage allocated Data Engineers, setting clear objectives, supporting development and wellbeing, and managing workload and delivery expectations. Coach and mentor engineers across the wider team, delegate meaningful technical ownership and build shared capability. * Remain hands-on in designing and developing complex pipelines, integrations and data models. Translate requirements into reusable, tested, production-ready solutions and provide technical direction and constructive review for other engineers. * Work with data owners, suppliers, analysts and governance and security colleagues to define data contracts, quality controls, access models, metadata, lineage and retention requirements. Implement appropriate engineering controls for secure and ethical data use and sharing, applying open standards and open-by-default principles where appropriate. * Embed disciplined engineering practices, including source control, automated testing, CI/CD, release management and environment configuration. Maintain clear architectural decisions, technical documentation and reusable implementation guidance. * Design for operational ownership with DataOps, including monitoring, structured logging, alerting, recovery and support runbooks. Lead resolution of complex technical issues, address root causes and manage technical debt, reliability, capacity and cost improvements.