> Markdown version of [/jobs/ext/1896544-azure-databricks-lead-engineer](https://www.wearedevelopers.com/jobs/ext/1896544-azure-databricks-lead-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Azure Databricks Lead Engineer

- **Company:** Horizontal Talent
- **Location:** Houston, TX, United States
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** Clean Code Principles, Business Logic, Microsoft Azure, Continuous Integration, Information Engineering, Data Governance, Data Infrastructure, Extract Transform Load (ETL), Data Systems, Operational Databases, SQL Databases, YAML, Enterprise Data Management, Data Lakes, Pyspark, Core Data, Deployment Automation, Data Management, Code Restructuring, Data Pipelines, Legacy Systems, Alteryx, Databricks
- **Published:** August 1, 2026
- **Apply:** https://www.dice.com/job-detail/08847807-6932-4f9b-b287-272b1438acde

## About the Role

* Strong hands-on experience with Azure Databricks in an enterprise environment.
 * Deep knowledge of Delta Lake and modern data engineering practices.
 * Proficiency in PySpark, SQL, and/or Scala.
 * Experience with Unity Catalog and Databricks governance capabilities.
 * Working knowledge of CI/CD practices and DevOps automation.
 * Experience building or supporting data platform migrations, especially from legacy systems.
 * Ability to translate business and ETL logic into clear, maintainable code.
 * Strong communication and collaboration skills.
 * Understanding of Azure fundamentals.

Preferred Skills

 * Databricks certifications, such as Data Engineer Associate or Professional.
 * Microsoft Azure Data Engineer Associate (DP-203) certification.
 * Microsoft Azure Fundamentals (AZ-900) or equivalent foundational Azure knowledge.
 * Experience converting Alteryx workflows into PySpark or SQL solutions.
 * Background supporting migration initiatives from Synapse into Databricks.
 * Experience working in hybrid team environments and coordinating across multiple stakeholders.

## Description

We're seeking an experienced Azure Databricks Lead Engineer to help shape and manage a modern enterprise data platform. This role is ideal for a hands-on data engineer who enjoys building scalable solutions directly in Databricks, leading migration efforts, and driving automation across data workflows.
Responsibilities

 * Design, build, and optimize end-to-end data pipelines within Azure Databricks and Delta Lake.
 * Develop solutions using PySpark, SQL, or Scala to support core data transformations and platform engineering.
 * Build and maintain production data workflows using Databricks Workflows and Jobs.
 * Lead migration efforts from legacy data platforms into Databricks-based solutions.
 * Refactor and convert existing business logic and ETL processes into scalable, code-driven Databricks notebooks.
 * Implement data governance practices using Unity Catalog, including access, security, schema management, and lineage.
 * Support CI/CD and DevOps automation for Databricks workspace deployments and jobs using Azure DevOps and YAML-based pipelines.
 * Collaborate with stakeholders to ensure data solutions are reliable, maintainable, and aligned with business needs.


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt)
- [CI/CD with Github Actions](https://www.wearedevelopers.com/videos/856-ci-cd-with-github-actions)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)

## Related Articles

- [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)
- [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)