Azure Databricks Engineer / Lead

We are seeking an experienced Azure Databricks Engineer / Lead to join our team in Dallas, TX. The ideal candidate will have strong hands-on experience designing and building scalable data engineering solutions using Azure Databricks, PySpark, Python, SQL, and Azure Data Services., * Design, develop, and implement scalable data pipelines using Azure Databricks.

• Build and optimize ETL/ELT pipelines using PySpark, Python, and SQL.
• Develop data solutions using the Medallion Architecture (Bronze, Silver, and Gold layers).
• Work with Delta Lake, Delta tables, and advanced Databricks optimization techniques.
• Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Azure Synapse, and Azure Key Vault.
• Develop and manage Databricks Workflows, Jobs, and Notebooks.
• Implement data quality, monitoring, error handling, and performance optimization.
• Collaborate with Data Architects, Data Engineers, Data Scientists, and business stakeholders.
• Establish best practices for CI/CD, Git/version control, code reviews, and automated deployments.
• Lead technical discussions and provide guidance to junior engineers.

• 8+ years of overall Data Engineering experience.
• Strong hands-on experience with Azure Databricks.
• Expert-level knowledge of PySpark and Spark SQL.
• Strong programming experience in Python and SQL.
• Experience with Delta Lake and Lakehouse architecture.
• Strong experience with Azure Data Lake Storage (ADLS Gen2).
• Experience with Azure Data Factory (ADF) and other Azure data services.
• Strong understanding of ETL/ELT, data modeling, and large-scale data pipelines.
• Experience with performance tuning and optimization in Databricks/Spark.
• Experience with Git, CI/CD, and DevOps practices.
• Excellent communication and stakeholder management skills.

Preferred Skills

• Databricks certifications.
• Experience with Unity Catalog and Databricks governance/security.
• Experience with Terraform or Infrastructure as Code.
• Knowledge of Azure DevOps.
• Experience working in a consulting or client-facing environment.