Senior Data Engineer / Databricks Specialist - US Citizens Only - Remote Role

• Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark and SQL to support data processing and analytics needs.
• Design, build and maintain robust ETL/ELT pipelines and implement Medallion Architecture (Bronze, Silver, Gold layers), ensuring data integrity, completeness and lineage.
• Administer Databricks workspaces, including cluster provisioning, job scheduling, access controls, Unity Catalog and notebook management.
• Manage AWS cloud infrastructure assets including S3, Lambda, IAM, Aurora and Glue, and provision environments using Terraform Infrastructure-as-Code (IaC).
• Perform query optimization, performance tuning, troubleshooting and debugging for complex Spark jobs and large-scale data solutions.
• Merge, consolidate and process structured, semi-structured and unstructured data sets from diverse API endpoints and databases.
• Embed security gates and compliance controls within DevSecOps CI/CD pipelines (Jenkins, GitLab CI) to support FISMA/FedRAMP requirements, ATO reassessments and POA&M resolution.
• Implement rigorous data quality controls, anomaly detection, validation rules and automated lineage tracking across all pipelines.
• Collaborate within cross-functional Agile/Scrum teams in multi-contractor federal environments, providing technical leadership and production on-call support.

Required experience and technical skills

• Citizenship: U.S. citizenship required.
• Mandatory certification: Active Databricks Certified Data Engineer Associate or Databricks Certified Data Engineer Professional.
• Experience: 6+ years of dedicated hands-on experience in Databricks development.
• Core big data stack: Production proficiency with PySpark, Spark SQL, Apache Spark and Delta Lake.
• Cloud ecosystem: Strong experience in AWS Cloud services including S3, IAM, Lambda, Glue and Aurora.
• Automation and DevOps: Production experience with Terraform (IaC) and CI/CD tools (Jenkins, GitLab CI or equivalent).
• Languages and frameworks: Deep proficiency in Python and SQL; working knowledge of REST APIs for ingestion.
• Clearance: Ability to obtain and maintain a Public Trust.
• Methodology and education: Hands-on experience in Agile/Scrum delivery environments. Bachelor s degree in CS, IT, Data Science or a related engineering discipline., * Hands-on experience with Unity Catalog, Delta Live Tables (DLT) and AI/ML pipeline support (MLflow, Model Registry).
• Experience in Federal Government contracting and compliance frameworks (FISMA / FedRAMP environments).
• Active security clearance (Public Trust or higher).

Preferred certifications In addition to the mandatory Databricks Data Engineer certification, candidates holding any of the following will be prioritized:

• Databricks Certified Machine Learning Associate
• AWS Certified Solutions Architect Associate or Professional
• AWS Certified Data Engineer Associate