Senior Data Engineer

We are seeking an experienced Senior Data Engineer / Data Engineer to design, build, and maintain robust data pipelines and architectures within NHS or Local Government environments. The postholder will ensure efficient data collection, storage, and access to support analytical, operational, and decision-making processes.

This role will work closely with data scientists, analysts, and operational teams to

ensure high-quality, secure, and accessible data across the organization.

Key Responsibilities

· Design, develop, and maintain scalable data pipelines, architectures, and ETL/ELT processes to support NHS or Local Authority data initiatives.

· Ensure efficient collection, storage, transformation, and access to both structured

and unstructured data from multiple sources.

· Collaborate with data scientists, analysts, and business stakeholders to enable analytics, reporting, and operational use of data.

· Implement data integration solutions that comply with NHS Digital standards, Data

· Security & Protection Toolkit (DSPT), and Local Authority policies.

· Ensure data pipelines and architecture adhere to UK GDPR, Data Protection Act 2018, and other relevant compliance and security requirements.

· Monitor, troubleshoot, and optimise data pipelines and infrastructure for performance, reliability, and scalability.

· Develop and maintain data documentation, metadata, and data quality standards.

· Advise on best practices for data storage, governance, and security in public sector contexts.

· Mentor and support junior data engineers and contribute to continuous improvement within the data engineering team.

Do you have experience in Spark?, · Proven experience as a Senior Data Engineer or similar role, ideally within NHS, Local Authority, or public sector environments.

· Strong expertise in ETL/ELT pipelines, data integration, and data architecture design.

· Proficiency with SQL, Python, or R for data manipulation and automation.

· Experience with data warehouses, data lakes, cloud platforms (Azure, AWS - (Glue, Redshift & EMR), GCP) and analytics tools.

· Knowledge of data governance, privacy, and security standards, particularly within NHS / public sector.

· Experience working with large, complex datasets and multiple data sources.

· Strong problem-solving skills and ability to work collaboratively with

· cross-functional teams.

· Experience mentoring junior staff or leading technical initiatives.

Desirable

· Experience with real-time or streaming data pipelines (e.g., Kafka, Spark Streaming).

· Knowledge of NHS Data Security & Protection Toolkit (DSPT) compliance requirements

· Familiarity with business intelligence tools such as Microsoft Fabric or Power BI, Tableau or Qlik.

· Professional certifications in cloud platforms, data engineering, or analytics., Degree in Computer Science, Data Engineering, Information Systems, or related field, or equivalent experience.

Relevant professional certifications desirable (e.g., Microsoft Certified: Azure Data

Engineer (DP-203), AWS Certified Data Engineer - Associate & AWS Certified Solutions Architect - Associate

Personal Attributes

· Highly analytical with strong attention to detail.

· Ability to translate technical solutions into business value.

· Excellent communication and stakeholder engagement skills.

· Proactive, collaborative, and adaptable to changing public sector priorities.