Data Engineer

The Home Office works to build a safe, fair and prosperous UK. We achieve this through our work on counter-terrorism, policing, crime, drugs policy, immigration and passports.

Home Office Digital designs, builds and develops services for the rest of the department and for government. Every year our systems support up to 3 million visa applications, checks on 100 million border crossings, up to 8 million passport applications and deliver 140 million police checks on people, vehicles and property.

Data Engineers implement the data analytics vision set by the Principal Data Engineer, designing and developing data flows to connect operational systems with analytics and BI systems. You’ll identify data sources, create ETL processes and collaborate with other teams to deliver the data strategy. Supporting Senior Data Engineers, you’ll help analyse problems and develop solutions. You’ll ensure data solutions are resilient, scalable and future-proof, working closely with the Data Science community to define project scope.

You’ll get to work with some of the largest and most varied datasets around, and benefit from a wealth of continuous professional development resources and career opportunities. You’ll play a key role in delivering joined-up, intelligent services that unlock the value from data and deliver better outcomes for the UK., Within the Central FinOps Team, the role focuses on enabling data products that help the Home Office understand, govern and optimise public cloud spend across AWS and Azure, including supporting showback, chargeback and cost optimisation outcomes. You will manage the ingestion, normalisation and enrichment of cloud billing and usage data, using CUR (including CUR2) and FOCUS datasets as key sources for reporting, and ensure cloud billing behaviour and commercial mechanisms are translated into trusted, reusable data assets.

Working closely with finance and engineering stakeholders, you will ensure outputs meet operational needs, including finance-ready reporting, while improving the quality of data underpinning allocation and recharge. The role also supports consistent access to insight through dashboards and datasets, including AWS QuickSight solutions enriched with organisational context.

Additionally, you will support problem analysis, contribute to FinOps standards, and work with the Data Services Science community to deliver user-driven data pipelines., your main day to day responsibilities will include.

• Assisting in the development of new data models and ETL processes, working collaboratively with the Analytics team to build data streaming systems

• Designing, building and testing data products based on feeds from multiple systems using a range of different storage technologies and/or access methods

• Applying the concepts and principles of data modelling to produce, maintain and update relevant data models for specific business needs, including reverse engineering data models from a live system

• Assisting in designing, writing, iterating and optimising code from prototype to production-ready

• Undertaking data profiling and source system analysis and presenting clear insights to colleagues to support the end use of the data

• Assisting in the successful delivery of completed data loads for customers, Data Analysts and Data Scientists, putting in place remedies and troubleshooting when required and assisting in the development of new data load programmes

• Overseeing the production/publication of regular and ad-hoc cloud costs reporting

• Recharging all public cloud costs across the department

• Working closely with Finance colleagues to maintain oversight of cloud spend across the organisation and complete accurate showback/recharging to ensure budgetary and regulatory compliance, answering ad-hoc finance-related cloud cost queries and contributing to business area budget decisions, The essential criteria listed above are reflective of the Home Office Government Digital and Data Profession Career Framework (based on the industry standard SFIA framework- Use the levels of responsibility to understand what would be expected for each technical skill listed). Please see below for the relevant skills for your role:

Strategy and Architecture

Strategy and Planning

Innovation (INOV) – Level 2

Advice and Guidance

Methods and Tools (METL) – Level 2

Development and Implementation

Data and Analytics

Data Management (DATM) – Level 2

Data Modelling and Design (DTAN) – Level 3

Database Design (DBDS) – Level 2

Data Engineering (DENG) – Level, If a person with disabilities is at a substantial disadvantage compared to a non-disabled person, we have a duty to make reasonable changes to our processes., * UK nationals

• nationals of the Republic of Ireland
• nationals of Commonwealth countries who have the right to work in the UK
• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities with settled or pre-settled status under the European Union Settlement Scheme (EUSS)
• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities who have made a valid application for settled or pre-settled status under the European Union Settlement Scheme (EUSS)
• individuals with limited leave to remain or indefinite leave to remain who were eligible to apply for EUSS on or before 31 December 2020
• Turkish nationals, and certain family members of Turkish nationals, who have accrued the right to work in the Civil Service

You’ll have a demonstrable passion for Data, with the following skills or some experience in:

• Programming using tools such as industry standard ETL tools, network databases and scheduling and orchestration tools, applying data development / engineering techniques – both in theory and practice or a strong aptitude to learn
• Effectively managing and communicating with a variety of stakeholders by translating technical concepts into non-technical language
• Managing the development and delivery of technical products
• Experience of using data engineering and analytical tooling used to process and analyse large-scale cloud cost and usage data, including SQL (Presto/Trino via Athena), PostgreSQL, and Python. This includes hands-on experience working with AWS Cost and Usage Reports (CUR v1/v2), FOCUS files, and semi-structured data formats such as JSON, Parquet and Iceberg, ensuring accuracy and consistency in financial reporting and analysis.
• An understanding of cloud data platforms, technologies, solutions and future Cloud Data Strategies and billing data models, particularly across AWS and Azure, including how usage, pricing, amortisation and allocation are represented. This includes knowledge of cloud provider billing intricacies, commitment-based discounts and incentives (e.g. Savings Plans, Reserved Instances, Azure Reservations, enterprise agreements), how they are applied in practice, and the known quirks and edge cases that affect cost transparency and optimisation.
• Proven ability to implement best practice for data engineering, automation and reliability, including version-controlled development (Git/GitLab or equivalent) and automated, repeatable pipelines. This includes designing and maintaining robust processes for cost ingestion, enrichment, validation and analysis, using services such as AWS Lambda, Step Functions, DynamoDB and Glue, and ensuring FinOps data products are auditable, resilient and fit for decision-making at scale.
• Experience in the use of cloud-native data technologies to deliver and evolve FinOps capabilities, including AWS and Azure, S3-based data lakes, Athena, Glue, Lambda and QuickSight (including dataset manipulation and analysis automation). This includes implementing Cloud Data and FinOps strategies to enable multi-cloud cost transparency, accurate allocation and recharge, and effective optimisation through a deep understanding of provider billing behaviour, discounts and commercial models, and an understanding of finance processes and procedures, including accounting and reporting., Experience in working with Big Data tools and data stores.

Technical skills

We'll assess you against these technical skills during the selection process:

• Innovation (INOV) – Level 2
• Methods and Tools (METL) – Level 2
• Data Management (DATM) – Level 2
• Data Modelling and Design (DTAN) – Level 3
• Database Design (DBDS) – Level 2
• Data Engineering (DENG) – Level 3, Successful candidates must undergo a criminal record check. Successful candidates must meet the security requirements before they can be appointed. The level of security needed is security check .

Pulled from the full job description

• Annual leave

• Company pension, £41,750 - £48,250 National pay locations: Salford - £41,750 - £44,250 London pay locations: Croydon - £45,750 - £48,250 New entrants to the Civil Service will start their role on the salary band minimum £41,750 for National Roles. Recruitment and Retention Allowance (RRA): Up to £5000 is available for candidates who demonstrate exceptional skills and experience evidenced at interview stage. GBP, In addition to your salary, a career with the Home Office offers a range of benefits, including:

• A Civil Service pension with an employer contribution of 28.97%.

• In-year reward scheme for one-off or sustained exceptional personal or team achievements.

• 25 days annual leave on appointment, rising with service to 30 days.

• Eight days of public holidays, plus one additional privilege day.

• Where business needs allow, some roles may be suitable for a combination of office and home-based working. This is a non-contractual arrangement where all employees will be expected to spend a minimum of 60% of their working time in an office.

• The advertised role is eligible for a Digital Capability Allowance. Successful candidates with exceptional skills and experience may apply for a Recruitment and Retention Allowance., The technical skills, have been ranked in order of importance to enable us to differentiate between candidates with tied interview scores

Candidates judged to be a near miss at interview may be offered a post at the grade below the one advertised without the need to go through a further selection process, providing the role has the same Success Profiles elements and essential skills. There is no guarantee that further such vacancies will arise.

Standards

All Civil Servants are expected to comply with the Civil Service Code, and its core values of integrity, honesty, objectivity and impartiality.

Additional Security Checks

As well as successfully obtaining UK Security Vetting clearance, candidates will be subject to a range of additional checks. The range of checks carried out will depend on whether you are a current Home Office employee, existing civil servant or an external new entrant as well as the role requirements. Further details are included in the Notes to Candidates.