Databricks Engineer-7

Do you have experience in YAML?, Minimum 6-8 years of relevant experience in below - Develop and maintain CI/CD pipelines for Azure Databricks deployments (Azure DevOps/YAML and related tools). Automate deployment and configuration of Databricks clusters, jobs, libraries, notebooks, and environment promotions. Implement and manage the Databricks environment for performance, cost efficiency, and scalability; optimize cluster sizing and autoscaling. Collaborate with Data Engineers/Scientists/Software Engineers to design, deploy, and scale data pipelines and models on Databricks. Monitor and troubleshoot clusters, pipelines, jobs, and associated workflows; integrate Azure Monitor/Log Analytics for visibility and metrics. Implement Infrastructure as Code (IaC) using Terraform, ARM templates, or Bicep to manage Azure resources and Databricks artifacts. Design and maintain backup, recovery, and DR strategies for Databricks environments. Support security best practices: RBAC/ABAC, managed identities, Key Vault secrets, compliance controls, and Unity Catalog governance. Produce clear documentation, templates, and runbooks; enable smooth KT to BAU teams. Proven experience as a DevOps/Platform Engineer in cloud environments, with a strong focus on Azure. Deep expertise in Azure Databricks, Azure Data Lake Storage, Azure Resource Manager (ARM), Microsoft Entra, Azure SQL Database. Hands on experience automating Databricks: clusters, libraries, jobs, notebooks, and environment promotions via pipelines. Proficiency in Unity Catalog and Databricks data governance. Familiarity with Apache Spark (PySpark, Spark SQL). Strong IaC skills: Terraform, ARM, or Bicep. Scripting (Python/PowerShell), and Git (branching strategies, conflict resolution). Observability with Azure Monitor, Log Analytics; pipeline orchestration with Azure Data Factory. Security best practices for cloud (RBAC, managed identities, Key Vault). Excellent problem solving and collaboration across cross functional teams.