> Markdown version of [/jobs/ext/1509912-data-engineer](https://www.wearedevelopers.com/jobs/ext/1509912-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Data Engineer - **Company:** 540 - **Location:** Arlington, VA, United States - **Experience:** Experienced - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Airflow, Amazon Web Services, Data Analysis, Computing Platforms, Automation of Tests, Microsoft Azure, Big Data, Cloud Database, Continuous Integration, Data Architecture, Data Discovery, Information Engineering, Data Governance, Extract Transform Load (ETL), Data Security, Document-Oriented Databases, Python (Programming Language), Metadata, Operational Databases, Cloud Services, Software Engineering, SQL Databases, Data Streaming, Unstructured Data, Workflow Management Systems, Data Processing, Google Cloud, Apache Spark, Data Lakes, Pyspark, Kubernetes, Infrastructure Automation Frameworks, Information Technology, Apache Kafka, Spark Streaming, Software Version Control, Data Pipelines, Docker, Databricks - **Published:** July 30, 2026 - **Apply:** https://www.clearancejobs.com/jobs/9066802/data-engineer ## About the Role Citizenship & Clearance Requirement: Per client requirements, candidates must be U.S. Citizens with an active DoW Secret (or higher) clearance Education Requirement: Bachelor's degree in Computer Science, Engineering, or a related technical field preferred; equivalent combinations of education and relevant experience will be considered 540 Internal Thrive Level: Data Engineer II or III, 4+ years of relevant data engineering or software engineering experience Hands-on experience developing and operating production data pipelines using Databricks Proficiency with Python, SQL, PySpark, Apache Spark, and Delta Lake Experience building automated ETL/ELT pipelines for large-scale datasets Experience designing and maintaining data models, schemas, tables, and lakehouse architectures Experience managing Databricks notebooks, jobs, workflows, and compute resources Experience implementing data quality, automated testing, monitoring, lineage, or metadata-management capabilities Experience working with Databricks and cloud-based data services in AWS, Azure, or Google Cloud Experience working with structured, semi-structured, and unstructured data Understanding of lakehouse architecture, data governance, security, privacy, and access-control principles Ability to troubleshoot data pipelines, Spark workloads, infrastructure, and applications NICE TO HAVE Databricks certification or equivalent demonstrated platform expertise Experience supporting DoW, federal, Advana, or other enterprise data environments Experience with CI/CD, infrastructure as code, automated testing, and source control Experience using Unity Catalog for data governance, lineage, and access control Experience developing streaming pipelines with Spark Structured Streaming, Kafka, Kinesis, or Pulsar Experience with orchestration tools such as Airflow, Dagster, or Argo Workflows Experience with Docker, Kubernetes, or other containerization and orchestration technologies Experience building cloud-native data platforms in secure, regulated, classified, or mission-critical environments Currently holds, or is willing to obtain within 30 days of employment, an approved certification such as Cloud+, GSEC, Security+, or SSCP ## Description 540 is seeking a Data Engineer to support a mission-critical technology modernization effort for the Department of War. You will design, build, and maintain Databricks-based data pipelines and lakehouse capabilities that enable secure data integration, analytics, AI/ML, and operational workloads at enterprise scale. Working with software engineers, AI/ML engineers, cybersecurity teams, and mission stakeholders, you will build scalable and reliable solutions using Databricks, Python, Apache Spark, and Delta Lake. The ideal candidate enjoys solving complex engineering challenges and developing trusted data products that support national defense missions., Design, develop, and maintain Databricks-based data pipelines, data products, and lakehouse capabilities Build automated ETL/ELT pipelines that ingest, transform, and deliver mission-critical data Develop production-grade data-processing solutions using Python, SQL, PySpark, Apache Spark, and Delta Lake Design and maintain data models, schemas, tables, and medallion architecture patterns supporting analytical, operational, and AI/ML workloads Build and operate batch and streaming data pipelines supporting mission requirements Develop and manage Databricks notebooks, jobs, workflows, clusters, and compute resources Implement data-quality checks, automated testing, monitoring, lineage, and metadata-management capabilities Support data discovery, governance, and access controls using Unity Catalog or similar technologies Optimize Spark workloads and Databricks resources for performance, scalability, reliability, and cost efficiency Collaborate with engineers, analysts, and data scientists to deliver reusable data products and mission capabilities Support Databricks deployments using CI/CD, infrastructure as code, and source control Partner with cybersecurity teams to implement data-protection, access-control, auditing, and governance requirements Troubleshoot issues affecting Databricks workloads, data pipelines, storage systems, and production data services Document data models, pipeline designs, engineering processes, and operational procedures ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [The Most Popular IT Jobs on the Market](https://www.wearedevelopers.com/magazine/376-the-most-popular-it-jobs-on-the-market)