Senior Azure DevOps Engineer

1. Infrastructure as Code (Primary)

• Design, implement, and maintain Terraform-based IaC for Azure and Databricks.
• Build reusable modules for multi-environment and multi-workspace deployments.
• Deploy and maintain Azure components:

• Databricks workspaces & metastores
• Unity Catalog policies
• Azure Data Lake Storage
• Networking, identity, and secure access patterns

• Implement policy-as-code, enforcing compliance, cluster policies, and guardrails.
• Ensure environments are secure-by-default, auditable, and automated end-to-end.

2. CI/CD Engineering (Primary)

• Design and operate CI/CD pipelines using Azure DevOps.
• Build CI/CD frameworks for Databricks assets (DAB):

• notebooks, packages, workflows, Delta Live Tables

• Implement automated validation: linting, tests, schema checks.
• Establish promotion strategies (dev * test * prod).
• Integrate Azure DevOps pipelines with Databricks Repos and enterprise Git workflows.

3. Platform Reliability & Operations

• Build monitoring and observability for Databricks workloads:

• job health, performance, usage, cost signals

• Implement alerting and dashboards for proactive operational management.
• Design systems for:

• safe rollback, retry strategies, idempotent operations

• Troubleshoot complex deployment and infrastructure issues.
• Improve performance, availability, and cost efficiency across the platform.

4. Governance, Security & Compliance

• Enforce enterprise governance via code:

• RBAC / ABAC (Unity Catalog)
• secret & credential management
• encryption, access patterns

• Align automation with architecture and governance requirements.
• Ensure security and compliance standards are embedded in CI/CD and IaC.

5. Cross-Functional Collaboration

• Partner with development, operations, data engineering, architecture, governance, and security teams.
• Act as a technical advisor and subject-matter expert for cloud and data platform automation.
• Reduce operational overhead by delivering self-service tools and automation., Milton Keynes with flexible remote working and occasional client site visits

We are seeking a highly skilled Senior DataOps / Azure DevOps Engineer to join our core data platform team. This role combines Azure CloudOps, Infrastructure as Code, and CI/CD engineering with a strong focus on platform reliability, security, and automation. The mission is to design, build, and operate a secure, scalable, and automated Databricks Lakehouse platform while ensuring enterprise-grade DevOps practices across Azure environments.

This is a senior individual contributor position with high technical ownership and cross-functional impact., * 8+ years of experience in DevOps, DataOps, or Platform Engineering

• Expert hands-on experience with Terraform and IaC design
• Strong experience with Azure Cloud and Azure DevOps CI/CD
• Deep understanding of Databricks platform operations
• Strong scripting skills: Python, Bash, or PowerShell
• Solid software engineering fundamentals: testing, modular code, GitOps
• Strong troubleshooting and problem-solving abilities
• Excellent communication & cross-team collaboration skills

Nice-to-Have

• Experience with Delta Lake, DLT, Databricks SQL
• Familiarity with Azure monitoring/observability tooling
• Exposure to streaming or event-driven architectures
• Experience supporting multi-environment enterprise platforms