Databricks Administrator

Virtualan Software LLC

United States

25 days ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

5 years minimum

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Amazon Web Services Amazon S3 Program Optimization Continuous Integration Data Architecture Information Engineering Data Governance Data Infrastructure Extract Transform Load (ETL) Data Security Data Systems Github

+20 more

Apache Hive Identity and Access Management Python (Programming Language) Operational Databases Performance Tuning Role-Based Access Control Standard Sql SQL Databases Apache Spark Infrastructure as Code (IaC) Data Lakes Pyspark Infrastructure Automation Frameworks Machine Learning Operations Cloud Optimization Cloudwatch Terraform Data Pipelines Amazon Redshift Databricks

Job description

We are looking for a Databricks Admin with strong hands-on experience in Databricks, AWS, and modern data platforms. The ideal candidate should have experience designing and supporting scalable data pipelines, administering the Databricks platform, optimizing performance and cloud costs, implementing governance, and troubleshooting production issues., * Design, build, and maintain scalable data pipelines using AWS and Databricks.

Develop ETL/ELT solutions using PySpark, Spark SQL, Delta Lake, Python, and SQL.
Create and manage Databricks clusters, SQL Warehouses, and cluster policies.
Configure and manage Unity Catalog, Metastore, RBAC, and data governance.
Implement Delta Sharing and support secure data access across teams.
Monitor and optimize Spark jobs, cluster performance, and Databricks workloads.
Drive cloud cost optimization (FinOps), including DBU optimization, tagging, budgets, and resource utilization.
Automate infrastructure and deployments using Terraform, Databricks Asset Bundles, GitHub Actions, or other CI/CD tools.
Work with AWS services such as S3, Glue, EMR, Lambda, Athena, Redshift, IAM, and CloudWatch.
Investigate and resolve production issues, perform root cause analysis, and improve platform reliability.
Collaborate with architects, developers, and business teams to deliver reliable data solutions.

Requirements

5+ years of Data Engineering experience in AWS.
Strong hands-on experience with Databricks, PySpark, Spark SQL, and Delta Lake.
Experience creating and managing Databricks clusters and platform administration.
Hands-on experience with Unity Catalog, Metastore, and Databricks governance.
Experience with Delta Sharing.
Strong Spark performance tuning and memory optimization skills.
Experience with AWS services including S3, Glue, EMR, Lambda, Athena, Redshift, IAM, and CloudWatch.
Strong Python and SQL skills.
Experience with cloud cost optimization (FinOps) and Databricks DBU optimization.
Experience with Terraform, Infrastructure as Code (IaC), GitHub Actions, or similar CI/CD tools.
Strong troubleshooting skills with experience supporting production data pipelines.

Preferred Skills

Databricks or AWS certifications.
Experience with MLflow, Delta Live Tables, or Databricks Asset Bundles.
Experience with AI-driven automation for platform operations.
Knowledge of modern Lakehouse architecture and cloud data governance., Candidates should be able to explain real-world experience with:
Databricks cluster creation and optimization.
Unity Catalog and Metastore setup.
Delta Sharing implementation.
Spark performance tuning and memory optimization.
Databricks cost (DBU) optimization and FinOps practices.
Terraform and CI/CD automation.
Resolving slow-running Databricks jobs.
Troubleshooting and resolving production data pipeline failures with root cause analysis.