Databricks SME

Agelix Consulting

Washington, United States of America

14 days ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Languages

English

Experience level

Senior

Compensation

$ 157K

Washington, United States of America

Amazon Web Services (AWS)

Cloud Computing

Code Review

Information Engineering

Data Governance

Identity and Access Management

Python

SQL Databases

Cloud Platform System

Amazon Web Services (AWS)

Data Lake

Data Lineage

Data Analytics

Data Pipelines

Databricks

Serve as the Databricks SME and guide technical decisions for workspace architecture design, configuration, and optimization.
Lead governance processes within Databricks environments, including access controls, data lineage, and workspace policies.
Optimize Databricks cluster and job performance; identify and resolve bottlenecks in data pipelines.
Design and implement secure integration patterns between Databricks and AWS services (e.g., S3, IAM, Glue, Redshift).
Perform specialized operations including Delta Lake management, Unity Catalog administration, and MLflow tracking.
Collaborate with government technical leads on architecture reviews and platform enhancement initiatives.
Develop and maintain technical documentation, runbooks, and best practice guidelines.
Support platform security requirements and ensure compliance with IRS/Treasury policies.
Mentor mid-level engineers and contribute to team knowledge sharing.
Participate in code reviews, design discussions, and agile ceremonies.

8+ years of hands-on experience in cloud technology environments.
3 5 years of dedicated Databricks experience including workspace administration and optimization.
Proven expertise as a Databricks SME with Unity Catalog, Delta Lake, and cluster management.
Strong proficiency with AWS services including S3, IAM, Glue, Redshift, and VPC networking.
Experience with data governance frameworks and security controls in cloud environments.
Proficiency in Python, Scala, or SQL for data engineering tasks.
Familiarity with federal IT compliance standards (FedRAMP, FISMA) preferred.
Knowledge of tax-processing or financial data management systems is highly preferred.
Databricks Certified Data Engineer Professional certification preferred.
AWS Certified Data Analytics or Solutions Architect certification preferred.