Data Pipeline Engineer

We are supporting a central government client who are seeking an experienced Data Pipeline Engineer and Architect to help design and build an open-source Data Transformation Layer for a national data exchange platform.

This is a hands-on engineering and architecture role, combining the design of reusable, reliable data pipelines with practical development using Apache NiFi, RDF, SPARQL, Python and Java.

Role and Responsibilities:

• Design and evolve the architecture of a reusable, cloud-agnostic data transformation platform.

• Build, test and maintain Apache NiFi flows, including integrations with APIs, databases, files and messaging systems.

• Develop pipelines to ingest, transform, validate and publish RDF triples, using SPARQL for querying and updates.

• Develop supporting components and integrations using Python and Java.

• Define data mappings, interface contracts, technical specifications and architectural decisions.

• Implement robust processing, including error handling, retries, duplicate management and recovery.

• Establish automated testing, monitoring, version control and repeatable deployment practices.

• Collaborate with architects, developers and stakeholders to translate requirements into working solutions.

• Produce clear technical documentation and support the delivery of reusable platform capabilities.

NB: Candidates must be eligible for BPSS security clearance which will processed following successful interviews (5-7 days).

• Proven experience designing and delivering data integration or ETL solutions using Apache NiFi.

• Practical project experience with RDF triples, SPARQL and semantic data modelling.

• Hands-on development experience in both Python and Java.

• Experience designing reusable pipelines across APIs, databases, files and other data sources.

• Understanding of reliable data processing, validation, automated testing and operational monitoring.

• Experience with version control, CI/CD, secure integrations and repeatable deployments.

• Ability to translate high-level requirements into technical designs and production-ready solutions.

Desirable Experience:

• Apache Jena and Jena Fuseki.

• Apache Kafka and event-driven data integration.

• SHACL or other RDF validation approaches.

• Custom NiFi processor development.

• Docker, Kubernetes and portable deployment architectures.

• Large RDF datasets, SPARQL optimisation or secure data exchange between organisations.