Senior Data Engineer

We are recruiting on behalf of a highly secure engineering and technology programme seeking an experienced Senior Data Engineer to support the design, development and delivery of secure, scalable and high-performance data solutions within a regulated environment.

This role will focus heavily on the Elastic Stack and Apache NiFi, supporting enterprise-scale data ingestion, transformation and analytics capabilities across complex secure systems.

The successful candidate will play a key role in architecting resilient data platforms, optimising real-time and batch processing pipelines, and ensuring compliance with strict governance and security standards.

KEY RESPONSIBILITIES

Data Architecture & Engineering

• Design and implement scalable, resilient data pipeline architectures using Elastic Stack and Apache NiFi

• Develop secure frameworks for data ingestion, transformation and integration

• Build and maintain batch and real-time data processing pipelines

• Ensure high standards of data quality, integrity and traceability across all systems

Elastic Stack & Platform Management

• Configure, optimise and maintain Elasticsearch clusters for performance and resilience

• Develop and maintain Logstash pipelines for efficient data processing

• Create dashboards and visualisations within Kibana to support operational and strategic reporting

• Monitor and manage large-scale data flows and platform performance in real time

• Carry out troubleshooting, root cause analysis and performance tuning activities

Security, Governance & Resilience

• Work closely with security and governance teams to ensure compliance with stringent security standards

• Implement secure data handling processes including encryption, RBAC and audit logging

• Support clustering, replication and failover strategies to ensure system availability

• Develop backup, disaster recovery and data retention strategies

Documentation & Collaboration

• Produce technical documentation covering architecture, workflows and security controls

• Support knowledge transfer and operational handovers to internal teams

Strong hands-on experience with:

• Elasticsearch
• Logstash
• Kibana

• Apache NiFi

• Proven experience designing and implementing enterprise-scale data pipelines

• Background working within secure or regulated environments such as defence, government or finance

• Strong understanding of:
• Data ingestion and processing frameworks
• Real-time and batch processing architectures
• Data modelling for analytics

• Distributed system troubleshooting and optimisation

• Knowledge of:
• Encryption and secure data handling
• RBAC and audit controls
• DevOps and CI/CD principles

• Git/version control systems

• Active DV Clearance essential

• Sole British National essential

DESIRABLE EXPERIENCE

• AWS, Azure or GCP within secure environments

• Docker and Kubernetes

• Kafka, Spark or additional data engineering technologies