> Markdown version of [/jobs/ext/1504265-senior-data-engineer](https://www.wearedevelopers.com/jobs/ext/1504265-senior-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer - **Company:** 540 - **Location:** Arlington, VA, United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Airflow, Amazon Web Services, Data Analysis, Automation of Tests, Microsoft Azure, Continuous Integration, Information Engineering, Data Infrastructure, Extract Transform Load (ETL), Data Security, Python (Programming Language), Metadata, Meta-Data Management, DataOps, Software Engineering, SQL Databases, Data Streaming, Google Cloud, System Availability, Apache Spark, Data Lakes, Pyspark, Infrastructure Automation Frameworks, Information Technology, Apache Kafka, Spark Streaming, Data Management, Machine Learning Operations, Terraform, Software Version Control, Data Pipelines, Databricks - **Published:** July 30, 2026 - **Apply:** https://www.clearancejobs.com/jobs/9066805/senior-data-engineer ## About the Role Citizenship & Clearance Requirement: Per client requirements, candidates must be U.S. Citizens with an active DoW Secret (or higher) clearance Education Requirement: Bachelor's degree in Computer Science, Engineering, or a related technical field preferred; equivalent combinations of education and relevant experience will be considered 540 Internal Thrive Level: Senior Data Engineer, 9+ years of relevant data engineering or software engineering experience Experience leading enterprise-scale data platform and pipeline implementations using Databricks Advanced proficiency with Python, SQL, PySpark, Apache Spark, and Delta Lake Experience architecting large-scale ETL/ELT, batch, and streaming pipelines Experience designing lakehouse architectures, data models, schemas, and data contracts Experience managing and optimizing Databricks jobs, workflows, compute resources, and Spark workloads Experience implementing data quality, monitoring, lineage, metadata management, and governance capabilities Experience with Unity Catalog or similar data-governance and access-control solutions Experience operating Databricks within AWS, Azure, or Google Cloud Experience with CI/CD, infrastructure as code, automated testing, and source control Strong understanding of data security, privacy, governance, and access-control principles Experience leading technical reviews, mentoring engineers, and communicating architecture decisions NICE TO HAVE Relevant Databricks certification Experience supporting DoW, federal, Advana, or other mission data environments Experience building data platforms in classified, regulated, or mission-critical environments Experience with Terraform, Databricks Asset Bundles, Airflow, or similar automation and orchestration tools Experience architecting streaming solutions using Kafka, Kinesis, Pulsar, or Spark Structured Streaming Experience supporting AI/ML pipelines, feature platforms, or MLflow Currently holds, or is willing to obtain within 30 days of employment, an approved certification such as Cloud+, GSEC, Security+, or SSCP ## Description 540 is seeking a Senior Data Engineer to support a mission-critical technology modernization effort for the Department of War. You will lead the design and evolution of Databricks-based data pipelines and lakehouse capabilities that enable secure data integration, analytics, AI/ML, and operational workloads at enterprise scale. Working with engineers, architects, cybersecurity teams, and mission stakeholders, you will translate complex requirements into secure, scalable solutions using Databricks, Apache Spark, and Delta Lake. You will define engineering standards, guide technical delivery, and mentor engineers while ensuring data quality, governance, and platform reliability., Lead the architecture and evolution of Databricks-based data pipelines and lakehouse capabilities Translate mission requirements into scalable data architectures and implementation strategies Define data engineering standards, reusable patterns, and best practices across engineering teams Architect automated ETL/ELT pipelines using Python, SQL, PySpark, Apache Spark, and Delta Lake Design scalable data models, schemas, data contracts, and medallion architecture patterns Lead the development of batch and streaming capabilities supporting operational, analytical, and AI/ML workloads Establish data quality, lineage, metadata, observability, and governance practices using Unity Catalog or similar technologies Optimize Databricks and Spark workloads for performance, scalability, reliability, and cost efficiency Establish CI/CD, infrastructure-as-code, testing, monitoring, and operational practices for Databricks environments Lead design reviews and resolve complex issues spanning data pipelines, infrastructure, and production services Partner with cybersecurity teams to incorporate security, access control, auditing, and governance requirements Communicate architecture decisions and mentor engineers on Databricks and data engineering best practices ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) - [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) ## Related Articles - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Best US AI Conferences for CTOs in 2026: Build vs. Buy, Vendor Evaluation, and Peer Intelligence](https://www.wearedevelopers.com/magazine/736-best-us-ai-conferences-for-ctos-in-2026-build-vs-buy-vendor-evaluation-and-peer-intelligence) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)