Azure Databricks Lead Engineer

Horizontal Talent

Houston, TX, United States

10 days ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Clean Code Principles Business Logic Microsoft Azure Continuous Integration Information Engineering Data Governance Data Infrastructure Extract Transform Load (ETL) Data Systems Operational Databases SQL Databases YAML

+11 more

Enterprise Data Management Data Lakes Pyspark Core Data Deployment Automation Data Management Code Restructuring Data Pipelines Legacy Systems Alteryx Databricks

Job description

We’re seeking an experienced Azure Databricks Lead Engineer to help shape and manage a modern enterprise data platform. This role is ideal for a hands-on data engineer who enjoys building scalable solutions directly in Databricks, leading migration efforts, and driving automation across data workflows. Responsibilities

Design, build, and optimize end-to-end data pipelines within Azure Databricks and Delta Lake.
Develop solutions using PySpark, SQL, or Scala to support core data transformations and platform engineering.
Build and maintain production data workflows using Databricks Workflows and Jobs.
Lead migration efforts from legacy data platforms into Databricks-based solutions.
Refactor and convert existing business logic and ETL processes into scalable, code-driven Databricks notebooks.
Implement data governance practices using Unity Catalog, including access, security, schema management, and lineage.
Support CI/CD and DevOps automation for Databricks workspace deployments and jobs using Azure DevOps and YAML-based pipelines.
Collaborate with stakeholders to ensure data solutions are reliable, maintainable, and aligned with business needs.

Requirements

Strong hands-on experience with Azure Databricks in an enterprise environment.
Deep knowledge of Delta Lake and modern data engineering practices.
Proficiency in PySpark, SQL, and/or Scala.
Experience with Unity Catalog and Databricks governance capabilities.
Working knowledge of CI/CD practices and DevOps automation.
Experience building or supporting data platform migrations, especially from legacy systems.
Ability to translate business and ETL logic into clear, maintainable code.
Strong communication and collaboration skills.
Understanding of Azure fundamentals.

Preferred Skills

Databricks certifications, such as Data Engineer Associate or Professional.
Microsoft Azure Data Engineer Associate (DP-203) certification.
Microsoft Azure Fundamentals (AZ-900) or equivalent foundational Azure knowledge.
Experience converting Alteryx workflows into PySpark or SQL solutions.
Background supporting migration initiatives from Synapse into Databricks.
Experience working in hybrid team environments and coordinating across multiple stakeholders.