Lead Data Engineer - Enterprise services

Home Office Digital designs, builds and develops services for the rest of the department and for government. Every year our systems support up to 3 million visa applications, checks on 100 million border crossings, up to 8 million passport applications and deliver 140 million police checks on people, vehicles and property.

Lead Data Engineers implement the Data Engineering vision set by the Principal Data Engineer. You’ll design and implement complex data flows, develop ETL processes and collaborate with teams to deliver data strategies. The role involves managing Data Engineering services, leading analytical projects and acquiring new data sources. You’ll work with various colleagues, including Data Architects, Database Administrators and Data Analysts, to establish and govern the enterprise Data strategy, ensuring alignment with organisational standards and objectives.

You’ll get to work with some of the largest and most varied datasets around, and benefit from a wealth of continuous professional development resources and career opportunities. You’ll play a key role in delivering joined-up, intelligent services that unlock the value from data and deliver better outcomes for the UK.

Where business needs allow, some roles may be suitable for a combination of office and home-based working. Where this is the case, employees will be expected to spend a minimum of 60% of their working time in the office. Applicants can raise any queries to the email address at the bottom of the advert.

Watch this short video to hear from members of Home Office Digital talking about the projects they work on and their experience of working here: Working for Home Office Digital., Lead Data Engineer x2 (DSA) - Data Services and Analytics (DSA) is the part of HO Digital which leads the build of data products to integrate data from multiple sources, and then deliver products for business intelligence, reporting and data science. DSA has the ambition to be one of the leading providers of insight services in UK Government and is currently growing significantly with over 300 colleagues, providing services to variety of functions within the Home Office.

These roles focus on delivering scalable cloud based data engineering by developing complex AWS data pipelines and high quality data models. With a large focus being DevOps automation with Terraform or AWS CloudFormation, Java based service support, Dynatrace monitoring integration, and managing CI/CD pipelines using Drone. You’ll also drive event driven architectures with Kafka and ensure reliable, well governed datasets that enable data driven decision making across the organisation.

Lead Data Engineer x 2 (EUC&C) – The End User Compute and Collaboration (EUC&C) team develops and delivers a range of Microsoft 365 solutions, including Teams, SharePoint, OneDrive, Power Platform, and Office applications. These tools support collaboration and productivity across the organisation.

We are establishing a new data capability to drive the adoption of data driven insights across the organisation’s product and service landscape. Working on end-to-end solutions in both AWS and Azure, using languages including Python, SQL, and PowerShell and orchestration platforms such as Glue and Fabric, you will lead the design, development, and maintenance of robust, scalable data pipelines. You’ll work closely with senior data colleagues and key stakeholders to shape and deliver EUC&C’s data vision., Your main day to day responsibilities will be:

• Managing data sources in a way to improve data quality and conform them for analysis.
• Developing new data models and ETL processes, working collaboratively with the analytics team and mapping data to an appropriate matching data model.
• Exercising key stakeholder management skills in order to manage expectations, listen and interpret business needs, communicate progress and facilitate difficult conversations, while building a 'service culture' around data.
• Managing the performance of suppliers and the data team to improve capability, as well as creating the roadmap which balances immediate user needs with long-term investment.
• Developing processes to balance data availability, capacity and data archiving, while continually improving processes and efficiency in the acquisition and ingestion of data.
• Line managing staff, including coaching and professional development., We will assess you against these technical skills during the selection process:

Strategy and Architecture

• Strategy and Planning

• Innovation (INOV) – Level 3

• Advice and Guidance

• Methods and Tools (METL) – Level 4

Development and Implementation

• Data and Analytics

• Data Management (DATM) – Level 4
• Data Modelling and Design (DTAN) – Level 4
• Database Design (DBDS) – Level 3
• Data Engineering (DENG) – Level 4, We'll assess you against these behaviours during the selection process, The allowance values are set by the Home Office, subject to remaining in a qualifying role, and are non-pensionable. This allowance is non-contractual, subject to an annual review and could be withdrawn at any time.

For both new entrants and existing civil servants, the total compensation offer is a combination of base salary and, if applicable, a capability-based allowance. The pay ranges for this role are National: £62,109 minimum- £68,320 maximum and London £66,229 minimum - £72,852 maximum. New entrants to the Civil Service will start on the pay range minimum. For existing civil servants, our policies on level transfer and promotion will apply.

Working at the Home Office

Every day, Home Office civil servants do brilliant work to develop and deliver policies and services that affect the lives of people across the country and beyond. To do this effectively and fairly, the Home Office is committed to representing modern Britain in all its diversity, and creating a welcoming, inclusive workplace where all our people are able to bring their whole selves to work and perform at their best.

We are flexible, skilled, professional and diverse. We work to recruit and retain disabled staff and area Disability Confident Leader. We are proud to be one of the most ethnically diverse departments in the civil service. We are a Social Mobility Foundation top 75 employer.

New entrants are expected to join on the minimum of the pay band.

Applicants who are successful at interview will be, as part of pre-employment screening, subject to a check on the Internal Fraud Database (IFD). This check will provide information about employees who have been dismissed for fraud or dishonesty offences. This check also applies to employees who resign or otherwise leave before being dismissed for fraud or dishonesty had their employment continued. Any applicant's details held on the IFD will be refused employment.

For further information please see the attached notes for candidates which must be read before making an application.

Existing Civil Servants should note that some of the Home Office terms and conditions of employment have changed. It is the candidate’s responsibility to ensure they are aware of the Terms and Conditions they will adopt should they be successful in application and should refer to the notes for candidates for further details.

Transfer Terms: Voluntary

You will need to meet the nationality requirements for this role and obtain the necessary security clearance to take it up.

Any move to the Home Office from another employer will mean you can no longer access childcare vouchers. This includes moves between government departments. You may however be eligible for other government schemes, including tax free childcare. Determine your eligibility at https://www.childcarechoices.gov.uk.

Reasonable Adjustments

If a person with disabilities is at a substantial disadvantage compared to a non-disabled person, we have a duty to make reasonable changes to our processes.

If you need a change to be made so that you can make your application, you should:

• Contact Government Recruitment Service via HOrecruitment.grs@cabinetoffice.gov.uk as soon as possible before the closing date to discuss your needs.
• Complete the “Assistance Required” section in the “Additional Requirements” page of your application form to tell us what changes or help you might need further on in the recruitment process. For instance, you may need wheelchair access at interview, or if you're deaf, a language service professional.

If you are experiencing accessibility problems with any attachments on this advert, please contact the email address in the ‘Contact point for applicants’ section.

Feedback Feedback will only be provided if you attend an interview or assessment.

Security

Successful candidates must undergo a criminal record check. Successful candidates must meet the security requirements before they can be appointed. The level of security needed is security check .

See our vetting charter . People working with government assets must complete baseline personnel security standard (opens in new window) checks., * UK nationals

• nationals of the Republic of Ireland
• nationals of Commonwealth countries who have the right to work in the UK
• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities with settled or pre-settled status under the European Union Settlement Scheme (EUSS)
• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities who have made a valid application for settled or pre-settled status under the European Union Settlement Scheme (EUSS)
• individuals with limited leave to remain or indefinite leave to remain who were eligible to apply for EUSS on or before 31 December 2020
• Turkish nationals, and certain family members of Turkish nationals, who have accrued the right to work in the Civil Service

Do you have experience in Terraform?, As a Lead Data Engineer in Enterprise Services you’ll have a demonstrable passion for Data, with the following skills or some experience in:

• Effectively leading and managing teams, helping them deliver technical products and projects in data analysis, solution design and end user reporting.
• Advising on best practice IT programming and automation techniques and tools such as GIThub, GITLab or other source control software, Python, Java.
• Leading the use of Cloud Data technologies, solutions and future Cloud Data Strategies with experience using tools such as Azure Automation, Azure Data Factory, Azure DevOps, ServiceNow, AWS Lambda, AWS Glue, Redshift, Athena, EC2, IAM, MS Fabric Notebooks and APIs such as REST, GraphQL.
• Working with stakeholders to gather requirements and deliver findings.
• Presenting complex ideas and recommendations in a clear concise way to technical and non-technical audiences, and senior customers.
• Using tools such as industry standard ETL tools, network databases and scheduling and orchestration tools such as Azure Automation, Azure Data Factory, Azure DevOps, AWS. Also, Step Functions, MS Fabric Task Flows, MSSQL, OneLake, AWS S3, AWS Redshift.

Desirable criteria for DSA roles

• Strong analytical and problem-solving skills including troubleshooting data issues and optimising data processes.
• Experienced in DevOps automation and cloud engineering, delivering scalable infrastructure using Terraform and/or AWS CloudFormation
• Hands-on experience integrating Dynatrace monitoring is advantageous.
• Skilled in Java, supporting Java-based services.
• Experience of managing CI/CD pipelines with Drone (or equivalent) and working with event-driven architectures using Kafka.

Desirable criteria for EUC&C roles

• Leveraging modern programming languages, such as Python and PowerShell, to develop and deliver high-quality data development and engineering solutions.
• Architecting and delivering robust conceptual, logical, and physical data models to underpin scalable, business-driven data platforms.
• Advising on best practice IT programming and automation techniques and tools.

SFIA capability framework

Skills for the Information Age (SFIA) is the technical framework that sets the standard capability and development of all levels in the Home Office. This is a link to the capability framework: All skills A - Z English (sfia-online.org).

We use set SFIA technical skills to form our interview questions and we will assess you against these technical skills during the selection process.

The essential skills listed above are reflective of the Home Office Government Digital and Data Profession Career Framework (based on the industry standard SFIA framework). Use the SFIA Levels of responsibility to understand what would be expected for each technical skills listed below., * Communicating and Influencing

Technical skills

We'll assess you against these technical skills during the selection process:

• Innovation (INOV) – Level 3
• Methods and Tools (METL) – Level 4
• Data Management (DATM) – Level 4
• Data Modelling and Design (DTAN) – Level 4
• Database Design (DBDS) – Level 3
• Data Engineering (DENG) – Level 4, This vacancy is using Success Profiles , and will assess your Behaviours, Experience and Technical skills.

As part of the application process, you will be asked to complete a CV and Personal Statement 1000 words). Please note your CV and personal statement should include all relevant experience that relates to our essential skills criteria listed in the advert and role description. We recommend that you use the STAR format in your examples and ensure that both components are completed thoroughly., Candidates reaching the required standard will then be invited to attend a final interview. The interview will assess your Technical Skills (SFIA Framework), Behaviours and Experience using technical and experience-based questions., For meaningful security checks to be carried out, individuals need to have lived in the UK for a sufficient period of time. Learn more on our website. Security Checks - Home Office Careers.

Please note that this role requires Security Check (SC) clearance, which would normally need 5 years’ UK residency in the past 5 years.

However, in exceptional circumstances security clearance applications for candidates who have been present in the UK for at least 3 of the last 5 years may be considered. Failure to meet this residency requirement will result in your security clearance application being rejected.

£62,109 New entrants to the Civil Service will start their role on the salary band minimum £62,109 for National or £66,229 for London roles. You may be eligible for an additional non-pensionable allowance, pending a Capability and Skills Assessment, with a value of up to £18,291. A Civil Service Pension with an employer contribution of 28.97% GBP, Alongside your salary of £62,109, Home Office contributes £17,992 towards you being a member of the Civil Service Defined Benefit Pension scheme. Find out what benefits a Civil Service Pension provides.

Why work for us...

Find out more information at: benefits - Home Office careers, but some of the primary ones are:

• A competitive starting salary.
• A civil service pension with employer contribution rates of at least 28.97%.
• In-year reward scheme for one-off or sustained exceptional personal or team achievements.
• The ability to potentially adopt flexible working options that suit your work/life balance, plus the opportunity in future to take a career break.
• 25 days annual leave on appointment, rising with service.
• Eight days public holidays, plus one additional privilege day.
• 26 weeks maternity, adoption or shared parental leave at full pay, followed by 13 weeks statutory pay and a further 13 weeks’ unpaid, after qualifying service.
• Maternity and adoption support leave (also known as paternity leave) of two weeks full pay, after qualifying service.
• Paid leave for fostering approval processes, support when a child is substantively placed with you plus a foster to adopt policy.
• Support for guardians and kinship carers.
• Corporate membership of ‘Employers for Carers’ providing additional information and advice for carers, plus a ‘Carer’s Passport’ to discuss workplace needs and underpin supportive conversations.
• Time off to deal with emergencies and certain other unplanned special circumstances., If you feel that your application has not been treated in accordance with the Civil Service recruitment principles and you wish to make a complaint, then contact the Government Recruitment Service via HOrecruitment.grs@cabinetoffice.gov.uk. If you are not satisfied with the response that you receive, then you can contact the Civil Service Commission.