> Markdown version of [/jobs/ext/3020986-senior-data-engineer](https://www.wearedevelopers.com/jobs/ext/3020986-senior-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer - **Company:** Omni Federal - **Location:** United States (Remote available) - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Kubernetes Security, Airflow, Amazon Web Services, Amazon S3, Apache HTTP Server, Code Review, Continuous Integration, Data as a Services, Data Validation, Information Engineering, Data Governance, Python (Programming Language), Operational Databases, Scrum Methodology, Software Factory, SQL Databases, Systems Integration, Apache Spark, Change Data Capture, Event Driven Architecture, Data Lakes, Debezium, Kubernetes, AWS Glue, Apache Kafka, Api Gateway, Terraform, Data Pipelines, Devsecops, Amazon Elastic Mapreduce (EMR) - **Published:** September 21, 2026 - **Apply:** https://www.clearancejobs.com/jobs/9180290/senior-data-engineer ## About the Role * 8+ years of professional data engineering experience, including hands-on design and operation of production data pipelines. * Demonstrated experience with streaming and batch data pipeline technologies, particularly Debezium (change-data-capture), Amazon MSK (Kafka), Apache Spark (AWS EMR), and Apache Airflow. * Strong proficiency in Python and SQL, with experience integrating and validating data from multiple heterogeneous sources. * Experience with AWS data services (e.g., S3, Glue, Aurora) and containerized environments (Kubernetes), delivered through DevSecOps/CI-CD pipelines. * Demonstrated ability to work independently against well-defined tasking with minimal oversight, and to communicate effectively with both technical and non-technical stakeholders, including data providers and consumers., * Experience supporting DoD or Intelligence Community programs, particularly Platform One or other DoD software factory environments. * Familiarity with container security and software supply chain concepts (SBOMs, vulnerability scoring and attestation). * Experience with the Apache Iceberg table format or similar open table formats (e.g., Delta Lake, Hudi) for lakehouse architectures. * Experience with event-driven architecture (e.g., AWS EventBridge) and Infrastructure-as-Code (Terraform). * Experience designing or consuming API Gateways in an enterprise data mesh architecture. * Prior experience in a vendor- or customer-facing data steward or data liaison capacity. Our team is always hungry for more - more knowledge, more problem solving, more growth, more innovation, and, ultimately, customer success. This hunger and hustle fuel our determination to excel in everything we do. It's not just about meeting established goals; it's about exceeding them. We take immense pride in applying our expertise in cutting-edge technologies and our ability to adapt to emerging trends directly to the users in the field, as our nation's protectors deserve nothing less. ## Description * Design, build, and operate streaming (Debezium change-data-capture into Amazon MSK) and batch (Apache Spark via AWS EMR) data pipelines that rapidly onboard new data sources into the program's Apache Iceberg-based data lakehouse, from initial ingestion through validation and production release. * Corroborate newly onboarded data against existing data lakehouse holdings to validate quality, consistency, and lineage before it is exposed to downstream consumers. * Develop and maintain the metadata, cataloging (AWS Glue as the Iceberg data catalog), and API Gateway integrations that make program data easily discoverable and consumable via Amazon Athena, Spark, and other Iceberg-compatible tools by downstream mission partners and vendors through the enterprise data mesh. * Own assigned technical tasking independently - translate well-articulated requirements into a working solution. * Engage directly with data providers and data consumers to understand the operational context behind each data source and translate that context accurately for both technical and non-technical audiences. * Participate in an agile (Kanban/Scrum) delivery team, contributing to sprint planning, technical design discussions, and pipeline/code reviews. * Maintain pipeline and infrastructure code as Infrastructure-as-Code (Terraform) and follow established DevSecOps build and deployment pipeline processes. * Support data governance, quality, and auditability practices in accordance with Government-approved data standards for the platform. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [How we built an AI-powered code reviewer in 80 hours](https://www.wearedevelopers.com/videos/1511-how-we-built-an-ai-powered-code-reviewer-in-80-hours) - [Python-Based Data Streaming Pipelines Within Minutes](https://www.wearedevelopers.com/videos/1233-python-based-data-streaming-pipelines-within-minutes) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [Tips, Techniques, and Common Pitfalls Debugging Kafka](https://www.wearedevelopers.com/videos/838-tips-techniques-and-common-pitfalls-debugging-kafka) - [AI Model Management Life Circles: ML Ops For Generative AI Models From Research to Deployment](https://www.wearedevelopers.com/videos/1152-ai-model-management-life-circles-ml-ops-for-generative-ai-models-from-research-to-deployment) ## Related Articles - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Fully Remote Software Engineer Jobs](https://www.wearedevelopers.com/magazine/447-fully-remote-software-engineer-jobs) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)