Senior Azure Databricks Platform Engineer

We are seeking an experienced Azure Databricks Engineer to join our team in Dallas, TX. The ideal candidate has strong hands-on experience in designing and building scalable data engineering solutions using Azure Databricks, PySpark, Python, SQL, and Azure Data Services - including leading the modernization and migration of existing Python object-oriented applications into scalable PySpark and Spark SQL data-processing solutions on Azure Databricks., * Design, develop, and implement scalable data pipelines using Azure Databricks.

• Build and optimize ETL/ELT pipelines using PySpark, Python, and SQL.
• Develop data solutions using the Medallion Architecture (Bronze, Silver, and Gold layers).
• Work with Delta Lake, Delta tables, and advanced Databricks optimization techniques.
• Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Azure Synapse, and Azure Key Vault.
• Develop and manage Databricks Workflows, Jobs, and Notebooks.
• Implement data quality, monitoring, error handling, and performance optimization.
• Collaborate with Data Architects, Data Engineers, Data Scientists, and business stakeholders.
• Establish best practices for CI/CD, Git/version control, code reviews, and automated deployments.
• Lead technical discussions and provide guidance to junior engineers.
• Analyze existing Python OOP applications and redesign single-node processing logic for distributed Spark execution.
• Design, develop, and deploy enterprise-scale data pipelines on Azure Databricks; build reusable PySpark frameworks and utility modules.
• Implement Delta Lake solutions using the Bronze, Silver, Gold architecture.
• Build robust ETL/ELT pipelines with Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics.
• Implement data quality, reconciliation, validation, and monitoring frameworks.
• Optimize Spark jobs using partitioning, bucketing, caching, broadcast joins, Adaptive Query Execution, and Delta optimization.

• 10 years of overall Data Engineering experience, including designing and implementing ETL/ELT pipelines with Azure Data Factory and other Azure services.

• Strong hands-on experience with Azure Databricks; 4 years of experience across Azure services and Databricks (ADLS, ADF, Azure DevOps, etc.).
• 7 years of Python development experience, with the ability to design and build reusable libraries.
• 4 years of experience with Snowflake or SQL (No-SQL experience is a plus).
• Expert-level knowledge of PySpark and Spark SQL.
• Strong programming experience in Python and SQL.
• Experience with Delta Lake and Lakehouse architecture.
• Strong experience with Azure Data Lake Storage (ADLS Gen2).
• Experience with Azure Data Factory (ADF) and other Azure data services.
• Strong understanding of ETL/ELT, data modeling, and large-scale data pipelines.
• Experience with performance tuning and optimization in Databricks/Spark.
• Experience with Git, CI/CD, and DevOps practices.

Preferred Skills

• Databricks certifications.
• Experience with Unity Catalog and Databricks governance/security.
• Experience with Terraform or Infrastructure as Code.
• Knowledge of Azure DevOps.
• Experience working in a consulting or client-facing environment.
• Experience designing APIs and integrating with React JS within a cloud platform.