> Markdown version of [/jobs/ext/2181655-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/2181655-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Data Engineer - **Company:** SHR CONSULTING GROUP, LLC - **Location:** Bluemont, VA, United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Automation of Tests, Cloud Database, Continuous Integration, Data Architecture, Information Engineering, Data Infrastructure, Data Stores, Software Debugging, Computer Data Storage, File Transfer, Python (Programming Language), Metadata, Performance Tuning, Standard Sql, Data Streaming, Systems Integration, Enterprise Data Management, Datadog, Data Processing, Data Ingestion, Apache Spark, Generative AI, Change Data Capture, Git, Data Lakes, Pyspark, Deployment Automation, Data Management, Video Streaming, Restful APIs, Software Version Control, Databricks - **Published:** August 22, 2026 - **Apply:** https://www.clearancejobs.com/jobs/9111641/lead-data-engineer ## About the Role * Extensive hands-on experience designing and developing enterprise data pipelines. * Demonstrated experience with Databricks or comparable cloud-based Lakehouse/data engineering platforms. * Strong experience with metadata-driven data ingestion frameworks. * Strong knowledge of schema evolution and enterprise data-processing patterns. * Demonstrated ability to troubleshoot live production pipeline failures. * Experience debugging complex data-storage and processing structures. * Experience designing batch and streaming pipelines. * Experience with cloud-based data engineering in Federal or similarly regulated environments. * Strong understanding of CI/CD, automated testing, data-quality validation, and deployment automation. * Experience with scalable data architectures and high-volume enterprise data environments. * Ability to provide hands-on technical leadership to engineering teams. Highly Desired Technical Experience * Databricks * Apache Spark / PySpark * SQL * Python * Delta Lake * Medallion/Lakehouse architectures * Structured Streaming or comparable streaming technologies * Cloud data services * REST APIs and source-system integration * Change Data Capture (CDC) * CI/CD and Infrastructure/Deployment Automation * Data quality and observability tooling * Git/version control * FinOps and cloud performance optimization Preferred Qualifications * Prior FEMA or DHS experience. * Experience migrating legacy Federal data environments to modern cloud data platforms. * Experience supporting data platforms used for AI/ML and Retrieval-Augmented Generation (RAG). * Experience engineering feature pipelines, curated AI datasets, vector/embedding data stores, or AI data products. * Experience with Federal security requirements including NIST, FISMA, FedRAMP, and DHS 4300A. Clearance Requirements * U.S. Citizenship. * Must be able to obtain and maintain the Government-required suitability/fitness determination and IT access authorization. ## Description * Lead technical delivery for FEMA pipeline engineering, operations, performance, and migration. * Design, construct, test, deploy, and maintain scalable batch and streaming data pipelines. * Build and maintain metadata-driven ingestion pipelines. * Implement robust schema evolution and data-processing patterns. * Diagnose and resolve production pipeline failures and complex data-storage issues. * Design pipelines consistent with FEMA's multi-zone/Medallion architecture. * Develop modular, parameterized, and reusable pipeline patterns. * Implement scalable ingestion and transformation solutions supporting increasing data volume, velocity, and concurrency. * Implement engineering practices for reliability and cost efficiency, including: + Incremental processing + Idempotent and restartable jobs + Partitioning + Storage/file-layout optimization + Compute right-sizing * Develop automated testing, data-quality validation, and deployment automation. * Diagnose schema drift, missing or late data, checkpoint/watermark failures, orchestration faults, and performance degradation. * Conduct root-cause analysis of significant or recurring production issues. * Support migration of legacy data, pipelines, reporting workloads, and dependencies. * Support source-system onboarding and integration using APIs, connectors, batch/file transfers, CDC, and custom integration patterns. * Provide technical leadership and mentoring to Senior Data Engineers and other engineering staff. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Debugging in the Dark](https://www.wearedevelopers.com/videos/1658-debugging-in-the-dark) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)