> Markdown version of [/jobs/ext/2593381-senior-data-engineer-on-site-washington-dc](https://www.wearedevelopers.com/jobs/ext/2593381-senior-data-engineer-on-site-washington-dc). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer (On Site, Washington, DC) - **Company:** Agile5 Technologies, Inc. - **Location:** Washington, DC, United States - **Experience:** Expert - **Salary:** $90,000.0 - $165,000.0 - **Contract:** Permanent contract - **Skills:** Agile Methodology, Amazon Web Services, Amazon S3, Apache HTTP Server, Business Logic, Audit Trail, Microsoft Azure, Cloud Engineering, Databases, Continuous Integration, Data Validation, Data Cleansing, Information Engineering, Data Governance, Extract Transform Load (ETL), Data Migration, Data Profiling, Github, Apache Hadoop, Apache Hive, Identity and Access Management, Job Scheduling, Python (Programming Language), Oracle (Applications), Role-Based Access Control, Power BI, Cloudera, Security Assertion Markup Language (SAML), SQL Databases, Systems Integration, Esri GIS (Software), Cloud Platform System, Okta, Informatica Powercenter, Change Data Capture, Amazon Relational Database Service, Data Lakes, Pyspark, Information Technology, Data Lineage, Data Pipelines, Legacy Systems, Jenkins, Databricks - **Published:** August 12, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=b9193fbc188f84cc ## About the Role * Minimum experience required varies by degree level: PhD with 4 years; Master's degree with 8 years; Bachelor's degree with 10 years; or High School Diploma with 14 years of relevant experience. * Hands-on experience with Databricks (workspace administration, notebook development, job scheduling) and proficiency in Python, PySpark, and SQL for large-scale pipeline development. * Experience migrating data from legacy platforms (Hive, Hadoop, Oracle) to cloud-native platforms, utilizing Delta Lake or Apache Iceberg table formats. * Practical experience with AWS services (S3, IAM, RDS, GovCloud), CI/CD pipeline tools (Azure DevOps, Jenkins, GitHub Actions), and enterprise ETL/ELT frameworks. Education Requirements: Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related technical field (or equivalent combination of education and experience). Desired Skills / Qualifications: * Databricks Certified Data Engineer (Associate or Professional). * Direct experience converting Informatica PowerCenter mappings to Python/PySpark code. * Experience with Unity Catalog data governance, Cloudera Hive/HiveQL, Power BI/ESRI integrations, and CDC methodologies. * Familiarity with FISMA High or FedRAMP High security requirements and SAML-based SSO/Okta integrations. ## Description Description: The Senior Data Engineer serves as the technical lead for complex data migration and pipeline engineering initiatives within enterprise data lakehouse modernization efforts. This role leads the conversion of legacy ETL artifacts into scalable Python/PySpark code, migrates database systems to cloud-native Delta Lake architectures, and implements robust data governance and automated reconciliation frameworks. Operating in a secure environment, the ideal candidate will manage Databricks infrastructure, design CI/CD pipelines, and mentor engineering personnel while ensuring complete data accuracy., * Serve as the primary technical lead executing daily migration operations, including data profiling, schema analysis, pipeline conversion, automated reconciliation, and production deployment validation. * Migrate legacy Hive tables to Delta Lake format on AWS S3 storage using Databricks ingestion tools. * Convert High and Medium complexity Informatica mappings to Python/PySpark code while preserving all business logic, data quality checks, and data cleansing routines. * Design and implement Change Data Capture (CDC) pipelines to maintain data synchronization between legacy and target environments during migration. * Configure and maintain Databricks workspaces, clusters, notebooks, and jobs within AWS GovCloud environments. * Implement Unity Catalog data governance, including RBAC, column-level encryption, audit trails, and data lineage tracking. * Build and execute automated data reconciliation scripts validating 100% migration accuracy. * Develop and maintain CI/CD pipelines for Python code deployment using Azure DevOps, checking in all converted code with comprehensive documentation. * Integrate the Databricks lakehouse with downstream applications (such as Power BI and ESRI) and configure SSO integrations. * Access legacy enclave environments for data profiling and side-by-side validation while collaborating daily with database managers and analysts. * Mentor mid-level engineering staff on platform operations and migration methodologies, participate in Agile ceremonies, and support system acceptance demonstrations. * Performs other duties as assigned. Security Clearance Requirements: * Public Trust / Tier 4 Eligible: No clearance required to apply; must be a U.S. citizen willing to undergo a background check to obtain a Public Trust / Tier 4 clearance. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [The Road to MLOps: How Verivox Transitioned to AWS](https://www.wearedevelopers.com/videos/1050-the-road-to-mlops-how-verivox-transitioned-to-aws) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) - [Our GitOps approach for deploying an Identity Provider and an API Gateway in a SaaS company](https://www.wearedevelopers.com/videos/776-our-gitops-approach-for-deploying-an-identity-provider-and-an-api-gateway-in-a-saas-company) - [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [The Most Popular IT Jobs on the Market](https://www.wearedevelopers.com/magazine/376-the-most-popular-it-jobs-on-the-market) - [Top-Paying Tech Jobs (with Salaries)](https://www.wearedevelopers.com/magazine/372-top-paying-tech-jobs-with-salaries)