Databricks Platform Engineer (Azure DevOps)

Platform & Infrastructure

• Design, deploy, and manage Databricks workspaces across multiple Azure subscriptions and environments (dev, staging, production)
• Implement Infrastructure-as-Code (IaC) solutions using Terraform for Databricks resources
• Establish and maintain security best practices, including network isolation, RBAC, and data governance
• Configure and Eligible to workimize cluster management, including auto-scaling policies and cost Eligible to workimization strategies
• Manage Databricks workspace connectivity and networking (VNets, private endpoints, firewalls)

DevOps & Automation

• Build CI/CD pipelines using Azure DevOps for deploying Databricks notebooks, jobs, and configurations
• Implement automated testing and validation of data pipelines
• Create self-service deployment frameworks and templates for development teams
• Establish monitoring, alerting, and logging solutions for Databricks environments
• Automate workspace provisioning and deprovisioning processes

Collaboration & Enablement

• Partner with data engineers and analysts to understand platform requirements
• Provide technical guidance and documentation for platform usage
• Develop runbooks and operational procedures for incident response
• Mentor team members on Databricks best practices and Azure DevOps tools

Governance & Compliance

• Implement access controls and audit logging across Databricks workspaces
• Ensure compliance with organizational security and data governance policies
• Monitor and manage workspace usage, costs, and resource utilization
• Document architecture decisions and maintain platform documentation

• 7 years of experience in platform engineering, infrastructure engineering, or DevOps
• 2 years of hands-on experience with Databricks infrastructure setup (or Apache Spark/Delta Lake environments)
• 3 years of experience with Azure cloud services and infrastructure
• 2 years of experience with Azure DevOps (Pipelines, Repos, Artifacts)
• Strong proficiency in Infrastructure-as-Code tools (Terraform Enterprise)
• Solid understanding of CI/CD principles and implementation
• Experience with Linux/Unix environments and scripting (Bash, Python, or similar)

• Familiarity with containerization technologies (Docker, Kubernetes)

• Strong knowledge of networking, security, and identity management in cloud environments