> Markdown version of [/jobs/ext/2474158-senior-data-engineer](https://www.wearedevelopers.com/jobs/ext/2474158-senior-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer - **Company:** Department for Environment, Food & Rural Affairs (DEFRA) - **Location:** Newcastle upon Tyne, UK (Remote available) - **Experience:** Expert - **Salary:** £42,665.0 - £50,495.0 - **Contract:** Permanent contract - **Skills:** Microsoft Azure, Cloud Database, Databases, Data Architecture, Information Engineering, Data Integrity, Extract Transform Load (ETL), Data Systems, Desktop Computing, Python (Programming Language), Meta-Data Management, Performance Tuning, SQL Databases, Data Streaming, Systems Integration, Cloud Platform System, Pandas, Pyspark, Data Lineage, Data Management, Azure Synapse Analytics, Data Pipelines, Databricks - **Published:** August 6, 2026 - **Apply:** https://uk.indeed.com/viewjob?jk=891a974a33af3a13 ## About the Role The ideal candidate will bring deep expertise in cloud-based data solutions, with hands-on experience developing and managing data platforms on technologies such as Azure and Databricks. They will excel at identifying opportunities to reuse and enhance existing workflows, optimising code to drive performance, scalability, and resilience. Equally comfortable engaging with both technical and non-technical stakeholders, they will act as a bridge between engineering teams, data consumers, and business leaders, ensuring data solutions are aligned with organisational goals., * Proven experience designing and building cloud-based data platforms that integrate large, complex, and disparate datasets - ideally using Databricks or similar technologies. * Advanced programming skills in Python (including PySpark, Pandas) and SQL for data engineering and analytics workloads. * Strong data modelling expertise, with a deep understanding of industry standards, best practices, and performance optimisation techniques. * Ability to lead the development of a portfolio of ETL pipelines, across a range of complex and/or large volume data. * Skilled at defining and implementing robust test conditions and validation frameworks based on business and technical requirements., * Outstanding communicator, capable of translating complex technical concepts into clear, actionable insights for technical teams and non-technical stakeholders alike. * Proactive and forward-looking, with a passion for horizon scanning and keeping up to date with emerging data tools, platforms, and best practices. * Strong problem-solving abilities — able to investigate complex data challenges, identify root causes, engage the right expertise, and deliver effective, sustainable solutions. * Strong team player with excellent collaboration skills, able to work effectively with colleagues to achieve shared goals., * Changing and Improving * Communicating and Influencing * Making Effective Decisions, * The technical assessment will include a range of questions and practical exercises designed to assess the technical skills, knowledge and problem-solving ability required for the role., This vacancy is using Success Profiles , and will assess your Behaviours, Experience and Technical skills., As part of the pre-employment process for this post, successful candidate(s) will be able to agree a contractual workplace from those locations listed in this advert. The agreed contractual workplace is then the substantive and permanent place of work for the successful candidate(s). Where the location is ‘National’ the successful appointee should discuss and agree an appropriate contractual location in line with both Defra’s location policy and site capacity, prior to proceeding with pre-employment processes. The agreed amount of time spent at a workplace for this post will reflect the requirement for Civil Servants to spend at least 60% of their working time in an organisation workplace with the option to work the remaining time flexibly from home. Working time spent at a workplace may include time spent at other organisational locations including field-based operational locations, together with supplier, customer or partner locations. This is a non-contractual agreement which is consistent with common Civil Service expectations. Travel costs to non-contractual workplaces will be subject to departmental travel and subsistence policies. Travel costs to contractual workplaces are the responsibility of the employee. The successful candidate is required to carry out all their duties from a UK location, and cannot do so from an overseas location at any time., Please take note that Defra does not hold a UK Visa & Immigration (UKVI) Skilled Worker License sponsor and are unable to sponsor any individuals for Skilled Worker Sponsorship. ## Description The Data and Analysis for Delivery (DA4D) division, part of Delivery and Group Corporate Services in Defra, plays a pivotal role in enabling data-driven decision-making across the department’s major delivery portfolio. Within DA4D, the Delivery Data team is responsible for the collection, management, and exploitation of data relating to the major delivery portfolio, performance and risk. By ensuring accurate, timely, and accessible data, the team supports Defra in understanding delivery progress and making informed strategic decisions. Its work underpins the department’s ability to manage complex programmes effectively and achieve outcomes that benefit the departments core purpose. You will play a key role in driving improvements in how Defra collects, manages, and harnesses data at enterprise level. Working closely with delivery teams, you will help identify, manage, and unlock the value of critical data assets, enabling evidence-based decision-making and more efficient operations. Your day-to-day work will be varied and impactful. One day you may be designing and scoping robust data architectures or modern storage solutions to enhance data quality, integrity, and accessibility across the organisation. On another, you might be developing and optimising data pipelines that deliver trusted, high-quality data to data scientists, analysts, and self-service users, supporting their need for timely and accurate insights. Collaboration will be at the heart of your role. You will work with a wide range of colleagues from across Defra, its arm’s length bodies, and external partners, optimising data flows, integrating with multiple data collection platforms, and ensuring data solutions are secure, scalable, and fit for purpose. Through your work, you will help shape and deliver Defra’s ambition to become a data-driven organisation., The team has recently delivered a critical milestone: an end-to-end Delivery Data Service, including a platform for collecting data, pipelines for ingesting and cleaning data, and a data science hub for analysis, visualisation, and interpretation. This foundation enables key services such as Portfolio Management, Enterprise Performance, and Enterprise Risk. Building on this, you will lead data engineering workstreams that enhance and scale our data architecture, optimise data flows, and deliver new pipelines to bring critical datasets into the team. Your focus will be on designing and implementing robust, cloud-based solutions that ensure data is accurate, accessible, and performant across the organisation. You’ll work with technologies such as Azure Synapse, Databricks, Python, and SQL to develop ETL processes, improve data quality, and enable downstream analytics and reporting. Collaboration will be central to your role. You’ll partner with colleagues across Defra group to integrate multiple data sources, streamline ingestion processes, and ensure solutions are secure, scalable, and aligned with best practices. This is your opportunity to shape how Defra manages and exploits data at enterprise scale, supporting one of the most complex delivery portfolios in government. You’ll thrive in a fast-paced, evolving environment - seeing opportunity in ambiguity, adapting quickly, and learning openly. As part of a small, ambitious, and friendly team, you’ll combine technical expertise with creativity to deliver meaningful change., * Design, build, and maintain cloud-based data platforms and architectures that integrate large, complex, and disparate datasets, ensuring scalability, security, and performance. * Develop and optimise ETL pipelines to ingest, transform, and deliver high-quality data for analytics, reporting, and operational use. * Implement robust data models and storage solutions that support efficient querying, data integrity, and enterprise-wide accessibility. * Collaborate with stakeholders across Defra group to understand data requirements, integrate multiple sources, and deliver solutions that meet business needs. * Ensure data quality and governance by applying validation frameworks, metadata management, and lineage tracking to maintain trusted datasets. * Stay current with emerging data engineering tools and best practices, contributing to innovation and continuous improvement within the team. * Lead technical workstreams and mentor colleagues, fostering a culture of collaboration, knowledge sharing, and engineering excellence., After submission of the first stage of your application you will be invited to complete a Civil Service Numerical Test. If you successfully pass the test, you will be invited to complete the final stage of the application. Please complete the online tests as soon as possible (within 24-48 hours is recommended), the closing date for the tests is 23:55 on 21/08/2026. If you fail to complete the online test before the deadline your application will be withdrawn. Guidance for the test will be available when you are invited to take the test. The tests are administered online and accessed via the CS Jobs website. Do not leave completing the tests until the last minute in case you experience any access or technical issues. There may not be technical support available after business hours and after 2pm on the closing date of 21/08/2026. While the tests should work on most operating systems and modern browsers with a good internet connection, we cannot guarantee that every combination will work. At the beginning, the test will check which browser you are using, and alert you if it is likely to be incompatible. It is highly recommended that you complete the online tests on a desktop computer, not a mobile phone or tablet. Please note that online tests associated with this vacancy are not subject to time limits; for further information visit the Reasonable adjustments for online tests - a candidate guide As part of the application process you will be asked to provide a CV, personal statement and evidence against the lead behaviour, Changing and Improving., If you pass Technical Assessment, you will be required to attend the final behaviour based interview. You will be assessed against the following three behaviours: * Behaviour 1 (Lead behaviour) Changing and Improving * Behaviour 2 Communicating and Influencing * Behaviour 3 Making Effective Decisions Interviews are expected to last no longer than 45 Minutes. Your interview will be conducted by video. Details of how to join your interview will be shared with you by the hiring manager., If a person with disabilities is put at a substantial disadvantage compared to a non-disabled person, we have a duty to make reasonable changes to our processes., Where more than one location is advertised, candidates will be appointed in merit order for each location. You will be asked to state your location preference on your application. Criminal Record Check If successful and transferring from another Government Department a criminal record check may be carried out. In order to process applications without delay, we will be sending a Criminal Record Check to Disclosure and Barring Service on your behalf. However, we recognise in exceptional circumstances some candidates will want to send their completed forms direct. If you will be doing this, please advise Government Recruitment Service of your intention by emailing Pre-EmploymentChecks.grs@cabinetoffice.gov.uk stating the job reference number in the subject heading. Internal Fraud Database Check Applicants who are successful at interview will be, as part of pre-employment screening, subject to a check on the Internal Fraud Database (IFD). This check will provide information about employees who have been dismissed for fraud or dishonesty offences. This check also applies to employees who resign or otherwise leave before being dismissed for fraud or dishonesty had their employment continued. Any applicant’s details held on the IFD will be refused employment. A candidate is not eligible to apply for a role within the Civil Service if the application is made within a 5 year period following a dismissal for carrying out internal fraud against government. Please be aware the levels of national security clearance are changing which may impact on the level needed for this role by the time of appointment. All efforts will be made to keep candidates informed of any changes and what that will mean in terms of vetting criteria. For more information please See our vetting charter For further information on National Security Vetting please visit the Demystifying Vetting website. Feedback Feedback will only be provided if you attend an interview or assessment. Security Successful candidates must undergo a basic (or equivalent) criminal record check. Successful candidates must meet the security requirements before they can be appointed. The level of security needed is counter-terrorist check . See our vetting charter . People working with government assets must complete baseline personnel security standard (opens in new window) checks., * UK nationals * nationals of the Republic of Ireland * nationals of Commonwealth countries who have the right to work in the UK * nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities with settled or pre-settled status under the European Union Settlement Scheme (EUSS) * nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities who have made a valid application for settled or pre-settled status under the European Union Settlement Scheme (EUSS) * individuals with limited leave to remain or indefinite leave to remain who were eligible to apply for EUSS on or before 31 December 2020 * Turkish nationals, and certain family members of Turkish nationals, who have accrued the right to work in the Civil Service ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [The 2026 Talent Pivot: Essential Trends for an Evolving Workforce](https://www.wearedevelopers.com/videos/100098-the-2026-talent-pivot-essential-trends-for-an-evolving-workforce) - [Advanced Typing in TypeScript](https://www.wearedevelopers.com/videos/496-advanced-typing-in-typescript) - [Kubernetes and Microservices with Multi-Model Databases](https://www.wearedevelopers.com/videos/382-kubernetes-and-microservices-with-multi-model-databases) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [UK Business Culture and Etiquette](https://www.wearedevelopers.com/magazine/326-uk-business-culture-and-etiquette) - [Software Engineer Salary in The UK](https://www.wearedevelopers.com/magazine/231-software-engineer-salary-in-the-uk) - [Fullstack Developer Salary UK](https://www.wearedevelopers.com/magazine/251-fullstack-developer-salary-uk) - [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london)