Lead Data Engineer

• Lead technical delivery for FEMA pipeline engineering, operations, performance, and migration.
• Design, construct, test, deploy, and maintain scalable batch and streaming data pipelines.
• Build and maintain metadata-driven ingestion pipelines.
• Implement robust schema evolution and data-processing patterns.
• Diagnose and resolve production pipeline failures and complex data-storage issues.
• Design pipelines consistent with FEMA’s multi-zone/Medallion architecture.
• Develop modular, parameterized, and reusable pipeline patterns.
• Implement scalable ingestion and transformation solutions supporting increasing data volume, velocity, and concurrency.
• Implement engineering practices for reliability and cost efficiency, including:
• Incremental processing
• Idempotent and restartable jobs
• Partitioning
• Storage/file-layout optimization
• Compute right-sizing
• Develop automated testing, data-quality validation, and deployment automation.
• Diagnose schema drift, missing or late data, checkpoint/watermark failures, orchestration faults, and performance degradation.
• Conduct root-cause analysis of significant or recurring production issues.
• Support migration of legacy data, pipelines, reporting workloads, and dependencies.
• Support source-system onboarding and integration using APIs, connectors, batch/file transfers, CDC, and custom integration patterns.
• Provide technical leadership and mentoring to Senior Data Engineers and other engineering staff.

• Extensive hands-on experience designing and developing enterprise data pipelines.
• Demonstrated experience with Databricks or comparable cloud-based Lakehouse/data engineering platforms.
• Strong experience with metadata-driven data ingestion frameworks.
• Strong knowledge of schema evolution and enterprise data-processing patterns.
• Demonstrated ability to troubleshoot live production pipeline failures.
• Experience debugging complex data-storage and processing structures.
• Experience designing batch and streaming pipelines.
• Experience with cloud-based data engineering in Federal or similarly regulated environments.
• Strong understanding of CI/CD, automated testing, data-quality validation, and deployment automation.
• Experience with scalable data architectures and high-volume enterprise data environments.
• Ability to provide hands-on technical leadership to engineering teams.

Highly Desired Technical Experience

• Databricks
• Apache Spark / PySpark
• SQL
• Python
• Delta Lake
• Medallion/Lakehouse architectures
• Structured Streaming or comparable streaming technologies
• Cloud data services
• REST APIs and source-system integration
• Change Data Capture (CDC)
• CI/CD and Infrastructure/Deployment Automation
• Data quality and observability tooling
• Git/version control
• FinOps and cloud performance optimization

Preferred Qualifications

• Prior FEMA or DHS experience.
• Experience migrating legacy Federal data environments to modern cloud data platforms.
• Experience supporting data platforms used for AI/ML and Retrieval-Augmented Generation (RAG).
• Experience engineering feature pipelines, curated AI datasets, vector/embedding data stores, or AI data products.
• Experience with Federal security requirements including NIST, FISMA, FedRAMP, and DHS 4300A.

Clearance Requirements

• U.S. Citizenship.
• Must be able to obtain and maintain the Government-required suitability/fitness determination and IT access authorization.

• Competitive salary commensurate with experience.
• Comprehensive medical, dental, and vision coverage.
• 401(k) with company contribution.
• Paid time off and eleven federal holidays.
• Certification reimbursement and training support.
• Life and disability insurance.

SHR Consulting Group, LLC is a small business delivering enterprise IT, cybersecurity, and program management services to the Department of Defense and federal civilian agencies. We support mission-critical infrastructure at the Pentagon and across the National Capital Region, and we invest in our people through competitive compensation, professional certification support, and long-term career growth on stable, multi-year programs.