Senior Data Engineer (Azure Databricks)

Our client is seeking a Senior Data Engineer with deep hands-on expertise in Azure Databricks to design, develop, and manage enterprise-scale data platforms. The ideal candidate will have strong experience with Databricks Workflows, Delta Lake, PySpark, Unity Catalog, Azure DevOps, and CI/CD automation. This role will focus on building code-driven data solutions, modernizing legacy data platforms, and migrating existing workflows into scalable Databricks-based architectures., * Design, develop, and optimize end-to-end data pipelines using Azure Databricks, Delta Lake, PySpark, SQL, and Scala.

• Build and manage production workflows using Databricks Workflows and Jobs for pipeline orchestration and automation.
• Lead migration efforts from legacy data platforms into modern Databricks environments.
• Convert legacy ETL workflows (including Alteryx workflows) into scalable PySpark/SQL-based solutions.
• Implement data governance, security, access controls, schemas, and lineage using Databricks Unity Catalog.
• Develop and maintain CI/CD processes for Databricks deployments using Azure DevOps, YAML pipelines, and automation frameworks.
• Optimize data processing performance, reliability, and scalability across enterprise data platforms., Need Azure Engineer Location: Houston ,TX/ Cleveland, OH Duration: 12 Months The primary purpose of this position is to provide technical leadership and engineering support fo…
• 28 days ago +

• Strong hands-on experience with Azure Databricks engineering and development.

• Expertise in building data pipelines using:

• PySpark
• SQL
• Scala
• Delta Lake
• Experience with Databricks Workflows and Jobs for production orchestration.
• Strong knowledge of Databricks Unity Catalog including governance, security, and lineage.
• Experience with Azure DevOps, CI/CD pipelines, YAML, and deployment automation.
• Experience migrating legacy data solutions into modern cloud data platforms.
• Strong understanding of enterprise data architecture and data engineering best practices.

Others Qualifications / Certifications

• Databricks certifications highly preferred:

• Databricks Certified Data Engineer Associate
• Databricks Certified Data Engineer Professional

• Microsoft certifications preferred:

• Azure Data Engineer Associate (DP-203)
• Azure Fundamentals (AZ-900) or equivalent Azure knowledge
• Experience converting Alteryx workflows into PySpark/SQL solutions is a major plus.
• Familiarity with legacy ETL modernization and reverse-engineering existing workflows.