> Markdown version of [/jobs/ext/2279223-tech-lead-data-ingestion-developer](https://www.wearedevelopers.com/jobs/ext/2279223-tech-lead-data-ingestion-developer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Tech Lead/Data Ingestion Developer - **Company:** Reply Inc - **Location:** United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Application Programming Interfaces (APIs), Airflow, Amazon Web Services, Amazon Elastic Compute Cloud, Amazon S3, Antivirus Softwares, Cloud Engineering, Cloud Storage, Information Engineering, Cryptographic Protocols, File Transfer, Monitoring of Systems, Identity and Access Management, Windows Servers, Reliability Engineering, Service Pack, Systems Integration, Software Vulnerability Management, Web Applications, Transport Layer Security, File Transfer Protocol (FTP), Cloud Platform System, Data Ingestion, Multi-Cloud, Technical Debt, Apache Nifi, Data Delivery, Cloudwatch, Data Pipelines, Servicenow - **Published:** August 28, 2026 - **Apply:** https://computerjobs.com/us/en/mob/job/51655088ED1788222B ## About the Role You are a motivated and adaptable professional with a strong analytical mindset and a passion for using technology to solve real-world problems. You enjoy working in collaborative, agile teams and take pride in delivering high-quality solutions that make a tangible impact. With strong communication skills and a consultative approach, you're comfortable engaging with clients, understanding their needs, and translating them into effective outcomes. ## Description We are seeking an experienced Tech Lead/Data Ingestion Developer to lead engineering efforts across a critical enterprise Data Ingest Service. The service operates as a trusted, secure, and resilient gateway responsible for ingesting, validating, scanning, and routing approximately half a million files monthly from both public and private sector data providers. Outputs are delivered across a hybrid ecosystem including on-premises Windows Servers, AWS S3, and Google Cloud Storage (GCS). This role balances hands-on technical leadership, operational resilience, and the strategic migration of Legacy architectures to a modern cloud-native stack., * Technical Leadership & Architecture: Lead the technical design, development, and delivery of secure ingest and egress data pipelines while defining standards for reliability, automation, and platform simplification. * Platform Modernisation: Drive the strategic transition from Legacy ingestion pipelines (Apache NiFi 1.x running on AWS EC2 clusters) toward a modernized stack leveraging Apache NiFi 2, AWS Step Functions, and AWS Managed Workspaces for Apache Airflow (MWAA). * Operational Excellence & Resilience: Establish a balanced operating model that separates live operational support from strategic engineering. Oversee incident resolution, root-cause analysis (RCA), monitoring/observability improvements, and automated platform recovery. * Security & Compliance: Ensure all incoming files undergo rigorous antivirus/malware scanning (eg, OPSWAT Metadefender integration) and security checks. Oversee vulnerability remediation, security patching, and SSL/TLS certificate renewals across the platform estate. * Stakeholder & Onboarding Support: Oversee technical work to onboard new survey and administrative datasets, reduce dataset onboarding times, and build self-service capabilities for internal and external data suppliers. * Team Leadership & Knowledge Sharing: Mentor engineers, eliminate single-point-of-failure knowledge bottlenecks through structured rotation, and foster cross-functional collaboration across network, security, and cloud platform teams. Technical Capabilities & Qualifications * Data Engineering & Orchestration: Deep expertise in Apache NiFi (v1.x and v2.x), AWS Step Functions, and Apache Airflow/AWS MWAA. * Cloud Architecture (AWS & Multi-Cloud): Hands-on experience with AWS core infrastructure (EC2, S3, IAM, CloudWatch) and multi-cloud data delivery targets (Google Cloud Storage, on-premises systems). * Security Integration: Demonstrated knowledge of integrating enterprise security controls, automated virus scanning solutions (eg, OPSWAT or similar), network security protocols, and secure file transfer (SFTP, API, Managed File Transfer). * Operational Engineering: Strong background in CI/CD pipelines, automated platform patching, infrastructure monitoring, and maintaining operational dashboards (eg, ServiceNow integration, alert tuning). * Web Applications & APIs: Experience supporting or integrating lightweight web applications that facilitate user-driven data ingestion. Core Deliverables & Strategic Objectives * Reduce Technical Debt: Proactively remediate vulnerabilities, automate platform maintenance, and eliminate recurring failure modes. * Accelerate Onboarding: Simplify data intake flows and automate processes to shorten the lead time for new dataset ingest requests. * Reliability Engineering: Transition the operating model from reactive symptom-fixing to preventative, long-term architectural stability. * Cross-Skilling: Build team resilience by documenting systems, establishing clear engineering standards, and rotating team members between live operational support and strategic projects. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Shipping Faster with Less: Render on Cloud Hosting, AI Workloads, and the Future of DevOps](https://www.wearedevelopers.com/videos/1894-shipping-faster-with-less-render-on-cloud-hosting-ai-workloads-and-the-future-of-devops) - [Applying Agile Principles to Incident Management ](https://www.wearedevelopers.com/videos/101-applying-agile-principles-to-incident-management) - [WeAreDevelopers LIVE - CSS is DOOMed](https://www.wearedevelopers.com/videos/1838-wearedevelopers-live-css-is-doomed) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [Why make use of an integration platform in today's software developments and infrastructure?](https://www.wearedevelopers.com/videos/758-why-make-use-of-an-integration-platform-in-today-s-software-developments-and-infrastructure) ## Related Articles - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers) - [Is Software Engineering Over-Saturated?](https://www.wearedevelopers.com/magazine/418-is-software-engineering-over-saturated) - [Top-Paying Tech Jobs (with Salaries)](https://www.wearedevelopers.com/magazine/372-top-paying-tech-jobs-with-salaries) - [Best Paying Jobs in Technology](https://www.wearedevelopers.com/magazine/256-best-paying-jobs-in-technology) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)