Data Pipeline Engineer

We are supporting a central government client who are seeking an experienced Data Pipeline Engineer and Architect to help design and build an open-source Data Transformation Layer for a national data exchange platform.

This is a hands-on engineering and architecture role, combining the design of reusable, reliable data pipelines with practical development using Apache NiFi, RDF, SPARQL, Python and Java.

The platform will use open-source, cloud-agnostic technologies to ingest, transform, validate and exchange data securely between systems., * Design and evolve the architecture of a reusable, cloud-agnostic data transformation platform.

• Build, test and maintain Apache NiFi flows, including integrations with APIs, databases, files and messaging systems.
• Develop pipelines to ingest, transform, validate and publish RDF triples, using SPARQL for querying and updates.
• Develop supporting components and integrations using Python and Java.
• Define data mappings, interface contracts, technical specifications and architectural decisions.
• Implement robust processing, including error handling, retries, duplicate management and recovery.
• Establish automated testing, monitoring, version control and repeatable deployment practices.
• Collaborate with architects, developers and stakeholders to translate requirements into working solutions.
• Produce clear technical documentation and support the delivery of reusable platform capabilities.

• Proven experience designing and delivering data integration or ETL solutions using Apache NiFi.
• Practical project experience with RDF triples, SPARQL and semantic data modelling.
• Hands-on development experience in both Python and Java.
• Experience designing reusable pipelines across APIs, databases, files and other data sources.
• Understanding of reliable data processing, validation, automated testing and operational monitoring.
• Experience with version control, CI/CD, secure integrations and repeatable deployments.
• Ability to translate high-level requirements into technical designs and production-ready solutions.

Desirable Experience:

• Apache Jena and Jena Fuseki.
• Apache Kafka and event-driven data integration.
• SHACL or other RDF validation approaches.
• Custom NiFi processor development.
• Docker, Kubernetes and portable deployment architectures.
• Large RDF datasets, SPARQL optimisation or secure data exchange between organisations.

Key Technologies:

• Contract
• Full Time

• Nationwide

• £475 - £525 / day
• 8d ago
• IT
• Work from home, * Contract
• Nationwide
• £490 - £510 / day

IT Work from home

Product Manager - National Surveillance Reporting Database 3-month contract (initially) Remote Working Outside-IR35 Market Rates (c£500 per day) The Opportunity: ISR Recruitment is seeking an experien… 3h ago, * Contract

• Nationwide
• £575 - £600 / day

Healthcare Work from home

Publications Lead (Medical Affairs) Remote Working (London + WFH) £575 to £600 per day 12-month contract (Inside IR35 / via Umbrella) The Opportunity: ISR Recruitment is currently supporting a global … 1d ago, * Contract

• Nationwide
• £550 - £575 / day

IT Work from home

Lead Java Software Engineer (Java / Spring Boot / Kafka / Kubernetes) 6 months (initially) Remote-working (UK-based) Market Rates (£550 to £575 per day) Outside IR35 The Opportunity: ISR Recruitment i… 2w ago