Databricks SME

Agelix Consulting

Washington, DC, United States

2 months ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

5 years minimum

Compensation

$78,000.0 - $157,000.0

Working hours

Regular working hours

Job source

careerjet.com

Tech stack

Amazon Web Services Amazon S3 Cloud Computing Code Review Information Engineering Data Governance Identity and Access Management Python (Programming Language) SQL Databases Cloud Platform System Amazon Virtual Private Cloud (VPC) Data Lakes

+4 more

Data Lineage Data Analytics Data Pipelines Databricks

Job description

Serve as the Databricks SME and guide technical decisions for workspace architecture design, configuration, and optimization.
Lead governance processes within Databricks environments, including access controls, data lineage, and workspace policies.
Optimize Databricks cluster and job performance; identify and resolve bottlenecks in data pipelines.
Design and implement secure integration patterns between Databricks and AWS services (e.g., S3, IAM, Glue, Redshift).
Perform specialized operations including Delta Lake management, Unity Catalog administration, and MLflow tracking.
Collaborate with government technical leads on architecture reviews and platform enhancement initiatives.
Develop and maintain technical documentation, runbooks, and best practice guidelines.
Support platform security requirements and ensure compliance with IRS/Treasury policies.
Mentor mid-level engineers and contribute to team knowledge sharing.
Participate in code reviews, design discussions, and agile ceremonies.

Requirements

8+ years of hands-on experience in cloud technology environments.
3 5 years of dedicated Databricks experience including workspace administration and optimization.
Proven expertise as a Databricks SME with Unity Catalog, Delta Lake, and cluster management.
Strong proficiency with AWS services including S3, IAM, Glue, Redshift, and VPC networking.
Experience with data governance frameworks and security controls in cloud environments.
Proficiency in Python, Scala, or SQL for data engineering tasks.
Familiarity with federal IT compliance standards (FedRAMP, FISMA) preferred.
Knowledge of tax-processing or financial data management systems is highly preferred.
Databricks Certified Data Engineer Professional certification preferred.
AWS Certified Data Analytics or Solutions Architect certification preferred.