Databricks Administrator

Virtualan Software LLC
yesterday

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

Remote

Tech stack

Amazon Web Services (AWS)
Amazon Web Services (AWS)
Program Optimization
Continuous Integration
Data Architecture
Information Engineering
Data Governance
Data Infrastructure
ETL
Data Security
Data Systems
Github
Hive
Identity and Access Management
Python
Operational Databases
Performance Tuning
Role-Based Access Control
Standard Sql
SQL Databases
Spark
Infrastructure as Code (IaC)
Data Lake
PySpark
Infrastructure Automation Frameworks
Machine Learning Operations
Cloud Optimization
Cloudwatch
Terraform
Data Pipelines
Redshift
Databricks

Job description

We are looking for a Databricks Admin with strong hands-on experience in Databricks, AWS, and modern data platforms. The ideal candidate should have experience designing and supporting scalable data pipelines, administering the Databricks platform, optimizing performance and cloud costs, implementing governance, and troubleshooting production issues., * Design, build, and maintain scalable data pipelines using AWS and Databricks.

  • Develop ETL/ELT solutions using PySpark, Spark SQL, Delta Lake, Python, and SQL.
  • Create and manage Databricks clusters, SQL Warehouses, and cluster policies.
  • Configure and manage Unity Catalog, Metastore, RBAC, and data governance.
  • Implement Delta Sharing and support secure data access across teams.
  • Monitor and optimize Spark jobs, cluster performance, and Databricks workloads.
  • Drive cloud cost optimization (FinOps), including DBU optimization, tagging, budgets, and resource utilization.
  • Automate infrastructure and deployments using Terraform, Databricks Asset Bundles, GitHub Actions, or other CI/CD tools.
  • Work with AWS services such as S3, Glue, EMR, Lambda, Athena, Redshift, IAM, and CloudWatch.
  • Investigate and resolve production issues, perform root cause analysis, and improve platform reliability.
  • Collaborate with architects, developers, and business teams to deliver reliable data solutions.

Requirements

  • 5+ years of Data Engineering experience in AWS.
  • Strong hands-on experience with Databricks, PySpark, Spark SQL, and Delta Lake.
  • Experience creating and managing Databricks clusters and platform administration.
  • Hands-on experience with Unity Catalog, Metastore, and Databricks governance.
  • Experience with Delta Sharing.
  • Strong Spark performance tuning and memory optimization skills.
  • Experience with AWS services including S3, Glue, EMR, Lambda, Athena, Redshift, IAM, and CloudWatch.
  • Strong Python and SQL skills.
  • Experience with cloud cost optimization (FinOps) and Databricks DBU optimization.
  • Experience with Terraform, Infrastructure as Code (IaC), GitHub Actions, or similar CI/CD tools.
  • Strong troubleshooting skills with experience supporting production data pipelines.

Preferred Skills

  • Databricks or AWS certifications.
  • Experience with MLflow, Delta Live Tables, or Databricks Asset Bundles.
  • Experience with AI-driven automation for platform operations.
  • Knowledge of modern Lakehouse architecture and cloud data governance., Candidates should be able to explain real-world experience with:
  • Databricks cluster creation and optimization.
  • Unity Catalog and Metastore setup.
  • Delta Sharing implementation.
  • Spark performance tuning and memory optimization.
  • Databricks cost (DBU) optimization and FinOps practices.
  • Terraform and CI/CD automation.
  • Resolving slow-running Databricks jobs.
  • Troubleshooting and resolving production data pipeline failures with root cause analysis.

Apply for this position