Senior Cloud Engineer (Azure)

We are seeking a professional to design, implement, and administer the Azure infrastructure behind our enterprise data and AI platform. This position is centered on Azure Databricks, ADLS Gen2, Azure Data Factory, and Azure Machine Learning. You will own the platform foundation - infrastructure as code, network and identity architecture, compute governance, observability, and cost management - and partner closely with our Databricks administration, data engineering, and data science teams. Part One - Infrastructure Ownership

• Design, implement, and administer the Azure foundation the platform runs on.
• Infrastructure as Code (Terraform-first): Author and maintain reusable Terraform modules.
• Network & Security Architecture: Design and implement private networking and security posture.
• Data Lake Infrastructure: Own ADLS Gen2 at the infrastructure layer.
• Data Factory Platform Operations: Own ADF from the platform side.
• AI/ML Platform Infrastructure: Provision and operate infrastructure supporting machine learning workloads.
• Observability & Reliability: Build and maintain the monitoring layer.
• FinOps & Capacity Planning: Own cost visibility and optimization.
• CI/CD & Environment Management: Own deployment pipelines and promotion paths.

Part Two - Databricks Platform Administration

• Day-to-day Databricks operations are supported by a dedicated administration team.
• Account & Workspace Administration: Account console configuration and workspace provisioning.
• Unity Catalog: Metastore design and regional strategy.
• Identity & Access: Entra ID integration and identity federation.
• Compute Governance: Cluster policies and instance pools.
• Cost & Usage Analysis: System tables and usage attribution.
• Technical Partnership: Review platform configurations and support administration and data teams.
• Cross-Boundary Diagnostics: Investigate and determine root causes of issues.

Role Scope

• Production ETL/ELT pipeline development and Spark transformation work sit with the data engineering team.
• Model development, training, and tuning sit with the data science team.
• Dimensional modeling, dbt development, and BI/semantic layer work sit outside this role.
• Day-to-day Databricks administration is handled by a dedicated team.

• Strong experience with Microsoft Azure Cloud
• Expertise in cloud infrastructure, deployment, and operations
• Hands-on experience with Azure services, networking, security, and automation
• Excellent communication and stakeholder management skills, * Experience: 6 years in cloud infrastructure or platform engineering, with at least 4 years focused on Microsoft Azure.
• Terraform: Expert-level, hands-on experience.
• Databricks Administration: Demonstrable hands-on administration experience.
• Technical Partnership: A track record as a senior technical resource to adjacent teams.
• Azure Networking & Security: Deep working knowledge of networking and security components.
• Data Lake Architecture: ADLS Gen2 design and access control.
• Azure Data Factory: Platform-side experience with integration runtimes and deployment automation.
• Automation & Scripting: Strong Python, plus PowerShell and/or Bash.
• Spark & SQL Literacy: Working knowledge to diagnose infrastructure and configuration-level performance issues., * Databricks Asset Bundles, the Terraform Databricks provider, and workspace-as-code patterns.
• Experience leading a Unity Catalog migration or multi-workspace consolidation.
• Azure Machine Learning, MLflow, or model-serving infrastructure experience.
• Kubernetes / AKS and containerized workloads.
• Policy as code - Azure Policy, OPA, Sentinel, or Checkov.
• Event-driven infrastructure - Event Hubs, Kafka, or Stream Analytics.
• Formal FinOps practice experience.
• Multi-region or multi-tenant Databricks deployments.

Preferred Certifications

• Databricks Certified Data Engineer Professional, or a Databricks platform administrator credential.
• Microsoft Certified: Azure Solutions Architect Expert (AZ-305).
• Microsoft Certified: Azure Administrator Associate (AZ-104).
• Microsoft Certified: DevOps Engineer Expert (AZ-400).
• HashiCorp Certified: Terraform Associate.