Tech Lead/Data Ingestion Developer

We are seeking an experienced Tech Lead/Data Ingestion Developer to lead engineering efforts across a critical enterprise Data Ingest Service. The service operates as a trusted, secure, and resilient gateway responsible for ingesting, validating, scanning, and routing approximately half a million files monthly from both public and private sector data providers.

Outputs are delivered across a hybrid ecosystem including on-premises Windows Servers, AWS S3, and Google Cloud Storage (GCS). This role balances hands-on technical leadership, operational resilience, and the strategic migration of Legacy architectures to a modern cloud-native stack., * Technical Leadership & Architecture: Lead the technical design, development, and delivery of secure ingest and egress data pipelines while defining standards for reliability, automation, and platform simplification.

• Platform Modernisation: Drive the strategic transition from Legacy ingestion pipelines (Apache NiFi 1.x running on AWS EC2 clusters) toward a modernized stack leveraging Apache NiFi 2, AWS Step Functions, and AWS Managed Workspaces for Apache Airflow (MWAA).
• Operational Excellence & Resilience: Establish a balanced operating model that separates live operational support from strategic engineering. Oversee incident resolution, root-cause analysis (RCA), monitoring/observability improvements, and automated platform recovery.
• Security & Compliance: Ensure all incoming files undergo rigorous antivirus/malware scanning (eg, OPSWAT Metadefender integration) and security checks. Oversee vulnerability remediation, security patching, and SSL/TLS certificate renewals across the platform estate.
• Stakeholder & Onboarding Support: Oversee technical work to onboard new survey and administrative datasets, reduce dataset onboarding times, and build self-service capabilities for internal and external data suppliers.
• Team Leadership & Knowledge Sharing: Mentor engineers, eliminate single-point-of-failure knowledge bottlenecks through structured rotation, and foster cross-functional collaboration across network, security, and cloud platform teams.

Technical Capabilities & Qualifications

• Data Engineering & Orchestration: Deep expertise in Apache NiFi (v1.x and v2.x), AWS Step Functions, and Apache Airflow/AWS MWAA.
• Cloud Architecture (AWS & Multi-Cloud): Hands-on experience with AWS core infrastructure (EC2, S3, IAM, CloudWatch) and multi-cloud data delivery targets (Google Cloud Storage, on-premises systems).
• Security Integration: Demonstrated knowledge of integrating enterprise security controls, automated virus scanning solutions (eg, OPSWAT or similar), network security protocols, and secure file transfer (SFTP, API, Managed File Transfer).
• Operational Engineering: Strong background in CI/CD pipelines, automated platform patching, infrastructure monitoring, and maintaining operational dashboards (eg, ServiceNow integration, alert tuning).
• Web Applications & APIs: Experience supporting or integrating lightweight web applications that facilitate user-driven data ingestion.

Core Deliverables & Strategic Objectives

• Reduce Technical Debt: Proactively remediate vulnerabilities, automate platform maintenance, and eliminate recurring failure modes.
• Accelerate Onboarding: Simplify data intake flows and automate processes to shorten the lead time for new dataset ingest requests.
• Reliability Engineering: Transition the operating model from reactive symptom-fixing to preventative, long-term architectural stability.
• Cross-Skilling: Build team resilience by documenting systems, establishing clear engineering standards, and rotating team members between live operational support and strategic projects.

You are a motivated and adaptable professional with a strong analytical mindset and a passion for using technology to solve real-world problems. You enjoy working in collaborative, agile teams and take pride in delivering high-quality solutions that make a tangible impact. With strong communication skills and a consultative approach, you’re comfortable engaging with clients, understanding their needs, and translating them into effective outcomes.

Solirius Reply, part of the Reply Group, is a technology consultancy and digital transformation partner that helps organisations solve complex challenges through strategy, design, engineering, and delivery.

We work closely with our clients to deliver secure, accessible, user-focused services that evolve with their needs. By combining deep technical expertise with people-centred design, we create solutions that deliver meaningful, lasting impact.

Our consultants partner directly with client teams, embedding into organisations to understand their goals, challenges, and users. This collaborative approach enables us to deliver tailored solutions that drive measurable outcomes across public and private sectors.

Past and present clients include the Ministry of Justice, Department for Education, Ministry of Housing, Communities and Local Government, UEFA, International Olympic Committee, and Mercedes-Benz. Our services span the full digital delivery life cycle, including architecture, engineering, delivery management, user-centred design, business analysis, data, DevOps, and AI.

We operate as a collaborative and inclusive organisation that empowers our people to take ownership, innovate, and develop their expertise. As an equal opportunities employer, we are committed to encouraging equality, diversity, and social mobility, while creating opportunities for our teams to work on meaningful projects that deliver lasting impact.