Databricks Engineer

Guidehouse is seeking a Databricks Engineer to join our AI and Data Defense and Security team to support client projects involving large-scale data pipelines, troubleshooting production workloads, and leading Agile delivery teams in support of organizational and mission goals within the Department of Homeland Security (DHS). This role requires hands-on technical expertise, strong communication skills, and the ability to own workstreams independently. Responsibilities include, but are not limited to:

• Design, build, and maintain scalable ETL pipelines using Databricks, PySpark, SQL, and Delta Lake.
• Proficiency in batch and stream processing, autoloader, and compute optimization.
• Troubleshoot and resolve Databricks job failures, cluster performance issues, data quality gaps, and pipeline bottlenecks.
• Implement CI/CD practices for notebooks, workflows, and data jobs.
• Lead Agile Project Management (stand-ups, sprint planning, retrospectives) as needed; support backlog refinement.
• Collaborate with cross-functional teams including data scientists, solution architects, and client stakeholders.
• Communicate technical concepts and delivery impacts to non-technical audiences.
• Ensure all engineering work aligns with data governance, security, and compliance standards (Unity Catalog, Delta constraints, Data Lineage).
• Perform code reviews, data quality validation, and documentation.

• Ability to OBTAIN and MAINTAIN a Federal or DHS Public Trust
• Bachelors Degree
• 5+ years of relevant experience

What Would Be Nice To Have:

• Active DHS Public Trust
• Law Enforcement experience
• Scrum Master Certification
• Experience managing projects in similar size and scope.
• Experience managing IT projects using either Agile Scrum or Kanban methodologies.
• Experience managing projects leveraging commercial cloud infrastructure
• Databricks Data Engineer Associate/Professional Certification.
• Experience with streaming frameworks (e.g., Kafka, Event Hub).
• Experience with CI/CD tools (Azure DevOps, Jenkins, GitHub Actions).
• Experience with Unity Catalog, Terraform, or related IaC tools.

Guidehouse offers a comprehensive, total rewards package that includes competitive compensation and a flexible benefits package that reflects our commitment to creating a diverse and supportive workplace.

Benefits include:

• Medical, Rx, Dental & Vision Insurance
• Personal and Family Sick Time & Company Paid Holidays
• Position may be eligible for a discretionary variable incentive bonus
• Parental Leave and Adoption Assistance
• 401(k) Retirement Plan
• Basic Life & Supplemental Life
• Health Savings Account, Dental/Vision & Dependent Care Flexible Spending Accounts
• Short-Term & Long-Term Disability
• Student Loan PayDown
• Tuition Reimbursement, Personal Development & Learning Opportunities
• Skills Development & Certifications
• Employee Referral Program
• Corporate Sponsored Events & Community Outreach
• Emergency Back-Up Childcare Program
• Mobility Stipend

About Guidehouse

Guidehouse is an Equal Opportunity Employer-Protected Veterans, Individuals with Disabilities or any other basis protected by law, ordinance, or regulation.

Guidehouse will consider for employment qualified applicants with criminal histories in a manner consistent with the requirements of applicable law or ordinance including the Fair Chance Ordinance of Los Angeles and San Francisco.