Lead Data Engineer

This role is central to DSIT’s ambition to unlock the full value of departmental data by replacing fragmented systems with a unified analytical tooling and data platform. By helping deliver this platform, you will directly enable analysts across the department to access tools and data quickly and securely. This will allow them to improve the evidence-base for key AI, science and emerging technologies policy decisions. This work will also provide the foundation for future AI tools, automation, and streamlined cross-team collaboration in DSIT.

The new platform will ensure analysts have:

• Timely, reliable access to high-value datasets
• Tools to carry out reproducible, secure, and efficient analysis
• Clear processes for procuring, ingesting, and sharing data
• A shared environment that reduces duplication and enables joined-up insights across teams.

Our design draws on the Ministry of Justice’s open-source Analytical Platform, widely recognised as a leading model across government, and has been adapted to meet DSIT’s needs as the digital centre of government.

A platform in two parts

The DSIT analytical platform is being delivered through two interconnected strands:

• Analytical tooling – delivering the secure, scalable tools analysts need, such as JupyterLab, RStudio, VS Code, Git integration, and cloud-based workflow orchestration.
• Data platform – delivering the pipelines, storage, and ingestion processes that ensure analysts can find and use trusted data efficiently.

Initially, this role will focus on strengthening and extending the Data platform, to meet the needs of analysts in DSIT. As the platform matures, you’ll also have opportunities to tackle broader Data Engineering challenges across the department, including data science and AI applications that improve operational efficiency and widen access to data for non-technical users.

Overall, the technical side of the platform team consists of two Lead DevOps Engineers, two Lead Data Engineers and three Software Developers. The team is led by a Product Owner, with support from a Technical Lead.

Timelines for the platform:

Phase 1: Establishing the core data architecture for ingestion, transformation and storage, setting the foundation for a scalable cloud-based platform that supports a wide range of data sources and use cases. This will already be complete.

Phase 2: User onboarding and refinement. You will join the project at this stage, helping to onboard a core group of analytical teams as early adopters. This will ensure they can easily access and use the data they need. Their feedback and insights will guide further improvements to the data architecture and help us prepare for a wider rollout across the department, including supporting teams with the migration of their existing datasets.

Where this role sits

The role is part of DSIT’s central Data Science and Engineering team, part of the department’s Analysis function. This team plays a key role in shaping DSIT’s analytical capability and works closely with technical colleagues in the Integrated Corporate Services (ICS) Digital team. By sitting within Analysis, the team ensures the platform is developed with analysts’ needs front and centre—owned by the analytical community and built to serve it.

As a G7 Data Engineer working on the Data platform, you will develop and extend the data infrastructure that underpins the entire analytical platform. You’ll establish scalable pipelines and processes, aligned with departmental data governance standards to make data assets accessible and reusable across analytical teams. Your work will contribute to the long-term vision of a single DSIT data system, ensuring the whole organisation is aligned on one trusted data source. This will directly improve evidence-based policymaking, unlock efficiency savings across the department, and create the foundations for future innovation and AI-enabled tools., * Design and deliver data pipelines to ingest, transform, and standardise datasets, to ensure discoverability across the department.

• Use cloud-native tools (e.g. AWS Glue, Lambda, Athena, S3) to develop scalable, maintainable data flows.
• Work closely with the other digital roles including DevOps Engineers and Software Developers to ensure effective delivery.
• Improve and maintain a centralised data store to enable joined-up analysis, reducing duplication and silos.
• Ensure pipelines are well-documented, tested, monitored and able to support user feedback and iteration.
• Work directly with analysts to understand their needs and translate them into robust data solutions., Candidates who meet the minimum benchmark may be placed on a Reserve List for consideration for similar roles, including those at a lower grade. Candidates who narrowly miss the benchmark and are not placed on the Reserve List may still be considered for an offer in a similar role at a lower grade.

Please note terms and conditions are attached. Please take time to read the document to determine how these may affect you.

Any move to the Department for Science, Innovation and Technology from another employer will mean you can no longer access childcare vouchers. This includes moves between government departments. You may however be eligible for other government schemes, including Tax Free Childcare; for further information visit the Childcare Choices website.

DSIT does not normally offer full home working (i.e. working at home); but we do offer a variety of flexible working options (including occasionally working from home).

DSIT cannot offer Visa sponsorship to candidates through this campaign.

DSIT holds a Visa sponsorship licence but this can only be used for certain roles and this campaign does not qualify., * UK nationals

• nationals of the Republic of Ireland
• nationals of Commonwealth countries who have the right to work in the UK
• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities with settled or pre-settled status under the European Union Settlement Scheme (EUSS)
• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities who have made a valid application for settled or pre-settled status under the European Union Settlement Scheme (EUSS)
• individuals with limited leave to remain or indefinite leave to remain who were eligible to apply for EUSS on or before 31 December 2020
• Turkish nationals, and certain family members of Turkish nationals, who have accrued the right to work in the Civil Service

Do you have experience in S3?, * Deep experience in cloud-based data engineering, ideally with AWS services.

• Strong Python skills for data processing (pandas, pyarrow, duckdb).
• Experience with data orchestration and pipeline frameworks (Airflow, dbt, AWS Glue Workflows).
• Sound knowledge of data modelling, standardisation and best practices in reproducible analytics.
• The ability to work closely with analysts, digital teams and fellow engineers to shape scalable, user-driven data infrastructure.

Desirable Criteria

• Familiarity with analytical tools (e.g. JupyterLab, RStudio, VS Code) and how analysts use them.

Technical skills

We'll assess you against these technical skills during the selection process:

• Communicating between technical and non-technical (practitioner)
• Data development process (expert)
• Programming and build (data and analytics engineering) (practitioner), Successful candidates must undergo a criminal record check. Successful candidates must meet the security requirements before they can be appointed. The level of security needed is security check .

Alongside your salary of £54,415, Department for Science, Innovation & Technology contributes £15,764 towards you being a member of the Civil Service Defined Benefit Pension scheme. Find out what benefits a Civil Service Pension provides.

The Department for Science, Innovation and Technology offers a competitive mix of benefits including:

• A culture of flexible working, such as job sharing, homeworking and compressed hours.
• Automatic enrolment into the Civil Service Pension Scheme, with an employer contribution of 28.97%.
• A minimum of 25 days of paid annual leave, increasing by 1 day per year up to a maximum of 30.
• An extensive range of learning & professional development opportunities, which all staff are actively encouraged to pursue.
• Access to a range of retail, travel and lifestyle employee discounts.

At DSIT we’re all about improving people’s lives by maximising the potential of science & technology. We accelerate innovation, investment and productivity through world-class science, research and development. We use technology for good by ensuring new and existing technologies are safely developed and deployed across the UK, with the benefits more widely shared. We are driving forward a modern digital government which gives citizens a more satisfying experience and their time back. We do all this to enable the Government’s 5 national missions: kickstarting economic growth, making Britain a clean energy superpower, taking back our streets, breaking down barriers to opportunity and building an NHS fit for the future. Above all, we focus on improving people’s lives. Whether it’s researching new treatments for disease, developing better batteries, reducing burdens through better public services, keeping children safe online, and much more, outcomes for citizens are at the heart of what we do.