Lead DevOps Engineer

The role is heavily AWS platform engineering focused. While Azure and DevSecOps experience are valuable, the day-to-day work is centred on designing, maintaining and improving AWS-hosted Identity platforms running on Kubernetes and ECS.

This is a hands-on technical leadership role. You will be expected to lead by example, mentor engineers, influence technical decisions and own services throughout their life cycle.

Core Technical Skills

AWS

Expected to have strong hands-on experience across AWS services including:

• ECS
• EKS
• EC2
• IAM
• Auto Scaling Groups
• KMS
• CloudHSM
• Secrets Manager
• Systems Manager Parameter Store
• CloudWatch

This is an engineering role rather than an infrastructure administration role.

Infrastructure as Code

Strong Terraform experience is expected.

This includes:

• Writing production-quality Terraform
• Creating and maintaining reusable Terraform modules
• Managing multiple environments using Terraform Workspaces
• Reviewing and improving existing modules
• Following engineering standards and code review practices

Kubernetes

A high level of Kubernetes knowledge is expected.

Daily work includes:

• Deploying and operating applications on Amazon EKS
• Troubleshooting workloads
• Working with Deployments, Services, Ingress, ConfigMaps and Secrets
• Understanding RBAC
• Resource requests and limits
• Horizontal Pod Autoscaling
• Network Policies
• Kubernetes troubleshooting

Helm

Helm is used extensively.

You will be expected to:

• Write Helm charts
• Maintain existing charts
• Build reusable templates
• Manage values across environments
• Debug Helm deployments

This is a core skill for the role.

ECS

A significant amount of the estate still runs on ECS.

Expected knowledge includes:

• ECS Services
• Task Definitions
• Auto Scaling Groups
• Capacity Providers
• Deployment strategies
• Operational troubleshooting

The team owns the ECS platform.

Docker

Almost every application is containerised.

Expected experience:

• Building images
• Writing Dockerfiles
• Multi-stage builds
• Image optimisation
• Container troubleshooting

GitLab & CI/CD

All source code is hosted in GitLab.

CI/CD is implemented using GitLab CI.

Expected experience includes:

• Writing GitLab CI pipelines
• Maintaining reusable pipeline fragments and templates
• Building and maintaining reusable CI/CD components
• Managing multi-stage deployment pipelines across Development, Test, Stage and Production
• Pipeline troubleshooting
• Secure secret handling
• Build and deployment automation

Testing & Quality Engineering

Testing is considered part of delivering a feature, not an optional activity.

Lead Engineers are expected to understand how to test infrastructure, platform code and applications, and to integrate those tests into automated delivery pipelines.

Expected experience includes:

• Writing and maintaining Terraform Tests
• Writing and maintaining Go-based tests for Helm charts where appropriate
• Writing or extending application tests (for example Node.js unit or integration tests)
• Writing automated tests for Python tooling and applications using frameworks such as pytest
• Understanding test coverage and identifying gaps
• Integrating automated tests into GitLab CI pipelines
• Troubleshooting failing automated tests
• Building reusable testing stages within GitLab CI templates
• Running tests locally and within CI/CD pipelines
• Performance, load and stress testing using JMeter and k6
• Integrating performance testing into GitLab CI pipelines where appropriate
• Analysing performance test results and identifying bottlenecks
• Producing repeatable automated validation rather than relying on manual testing

Testing is expected to form part of the Definition of Done.

Identity & Security

As an Identity programme, understanding authentication is important.

Expected knowledge includes:

• OAuth 2.0 flows
• OpenID Connect (OIDC)
• JWTs
• API authentication
• IAM
• Kubernetes RBAC

Knowledge of identity platforms and authentication patterns is highly desirable.

API Gateway

The platform makes extensive use of Kong API Gateway.

Useful experience includes:

• Routing
• Authentication plugins
• OIDC integration
• Rate limiting
• Upstream configuration
• Troubleshooting gateway behaviour

Service Mesh

Applications running on EKS use Istio.

Expected knowledge includes:

• Virtual Services
• Destination Rules
• Gateways
• mTLS
• Traffic routing
• Basic troubleshooting

Monitoring & Observability

Current monitoring stack:

• Prometheus
• Grafana

The organisation is introducing Dynatrace, either alongside or as a future replacement.

Expected experience includes:

• Creating dashboards
• Alerting
• Metrics
• Performance troubleshooting
• Root cause analysis
• Monitoring production services

Technical Documentation

Strong technical documentation is an expected part of the role.

You will be expected to:

• Produce clear Spike documentation covering the problem, investigation, options considered, recommendation and outcome
• Write implementation documentation for new features and architectural changes
• Maintain comprehensive README files for repositories, Terraform modules and Helm charts
• Produce operational runbooks
• Create support documentation
• Produce handover documentation where appropriate
• Record technical decisions and their reasoning
• Communicate ideas and proposals in a structured way
• Write high-quality Jira stories, technical tasks and Spike tickets with clear acceptance criteria, implementation detail and expected outcomes

Good written communication is considered a core engineering skill.

Technical Ownership

Lead Engineers are expected to take ownership of services rather than simply deliver individual pieces of work.

This includes:

• Owning services throughout their life cycle, from design through to production support
• Understanding the wider platform and the impact of technical decisions
• Driving work forward with minimal supervision
• Identifying technical debt and proposing practical improvements
• Challenging designs where there are operational, security, resilience or scalability concerns
• Considering operational support as part of every solution
• Designing solutions that are maintainable by the wider engineering team
• Working across multiple repositories and services
• Being accountable for the quality of technical delivery
• Supporting incident investigation and root cause analysis through to completion

Agile Delivery & Ways of Working

The team follows Agile delivery practices and Lead Engineers are expected to actively contribute to both technical delivery and the running of the team.

Expected experience includes:

• Working effectively within Agile delivery teams
• Participating in Sprint Planning, Sprint Reviews, Retrospectives and Daily Stand-ups
• Leading Agile ceremonies when required, with all engineers taking turns facilitating stand-ups, retrospectives and sprint planning
• Breaking down technical work into deliverable stories and tasks
• Writing high-quality Jira stories, technical tasks and Spike investigations
• Producing clear acceptance criteria and technical implementation plans
• Contributing to backlog refinement by helping define technical requirements and identify dependencies
• Providing realistic estimates and identifying technical risks early
• Collaborating closely with Product Owners, Delivery Managers, Architects and other engineering teams
• Managing priorities while balancing feature delivery, technical debt, operational support and platform improvements
• Communicating progress, blockers and technical decisions clearly
• Promoting continuous improvement across engineering practices and delivery

Lead Engineers are expected to help create an environment where knowledge is shared, engineering standards are maintained and the team continues to improve its ways of working.

Leadership Expectations

This is a Lead DevOps Engineer position.

Technical leadership is as important as technical delivery.

Expected behaviours include:

• Supporting and mentoring permanent DWP engineers through coaching, pair programming and technical guidance
• Providing technical direction within the squad
• Leading engineering discussions and helping the team reach sound technical decisions
• Making pragmatic technical decisions within agreed architectural standards
• Reviewing designs, code and merge requests, providing constructive feedback
• Identifying opportunities to improve existing processes, tooling and automation within approved technology boundaries
• Maintaining engineering standards across the team
• Producing clear technical documentation to support delivery and future maintenance
• Communicating technical ideas, risks and recommendations to engineers, architects and stakeholders
• Working closely with Product Owners, Delivery Managers and Architects to shape technical delivery
• Influencing delivery through technical expertise rather than authority
• Owning technical problems through to resolution
• Setting a high engineering standard through the quality of your own work

The environment is governed by established technology standards, so improvements are generally expected within the approved tooling and architecture rather than introducing alternative products or platforms.

Daily Activities

Typical work includes:

• Developing infrastructure using Terraform
• Maintaining reusable Terraform modules
• Developing and maintaining Helm charts
• Supporting and evolving reusable Helm chart libraries
• Deploying services to ECS and EKS
• Supporting the ECS platform
• Writing and maintaining GitLab CI pipelines and reusable pipeline templates
• Writing and maintaining automated tests for infrastructure, platform code and supporting applications
• Integrating automated testing into GitLab CI pipelines
• Investigating and resolving test failures
• Performance, load and stress testing using JMeter and k6
• Supporting production services
• Investigating incidents and performing root cause analysis
• Reviewing merge requests
• Participating in technical design discussions
• Writing and maintaining technical documentation
• Producing Spike investigations and implementation proposals
• Writing Jira stories, technical tasks and Spike tickets
• Improving automation and engineering standards
• Working with application developers to support deployments
• Maintaining AWS platform components
• Mentoring engineers
• Leading Agile ceremonies when required
• Driving continuous improvement across the platform and engineering practices

Technologies Used

• AWS
• ECS
• EKS
• EC2
• Terraform
• Terraform Tests
• Helm
• Go
• Kubernetes
• Docker
• GitLab
• GitLab CI
• Kong API Gateway
• Istio
• OAuth 2.0
• OpenID Connect (OIDC)
• JWT
• IAM
• RBAC
• CloudHSM
• AWS KMS
• Secrets Manager
• Systems Manager Parameter Store
• Prometheus
• Grafana
• Dynatrace
• Python
• pytest
• Node.js
• JMeter
• k6

What Success Looks Like

After approximately three months in the role, a successful Lead DevOps Engineer would be expected to:

Technical Delivery

• Be comfortable deploying, supporting and troubleshooting services running on ECS and EKS
• Deliver production-ready Terraform, Helm and GitLab CI changes with minimal supervision
• Understand the architecture of the Identity platform and how the various services interact
• Be comfortable working across multiple repositories, environments and AWS services
• Successfully troubleshoot production issues and participate in root cause analysis
• Write well-tested code, infrastructure and automation with appropriate automated validation
• Confidently work with Kubernetes, Docker, Kong API Gateway, Istio and supporting AWS services
• Produce secure, maintainable and supportable engineering solutions

Engineering Quality

• Produce reusable, well-structured Terraform modules and Helm charts
• Contribute improvements to shared GitLab CI templates and reusable pipeline components
• Maintain high standards of testing across infrastructure, tooling and application code
• Integrate automated testing into CI/CD pipelines as part of normal delivery
• Write clear, maintainable and well-documented code
• Leave services and repositories in a better state than they were found

Documentation

• Produce high-quality Spike investigations with clear recommendations and outcomes
• Write clear Jira stories, technical tasks and Spike tickets with meaningful acceptance criteria
• Maintain accurate README files and implementation documentation
• Produce documentation that enables other engineers to understand, support and extend the platform
• Clearly communicate technical decisions and the reasoning behind them

Leadership

• Act as a technical leader within the squad
• Support and mentor less experienced engineers and permanent DWP staff
• Lead engineering discussions and help drive technical decisions
• Provide constructive feedback through code reviews and technical discussions
• Identify opportunities for improvement and help prioritise engineering effort
• Take ownership of services and technical problems through to completion
• Be recognised as someone other engineers naturally approach for technical guidance

Agile Delivery

• Actively contribute to Sprint Planning, Reviews, Retrospectives and Daily Stand-ups
• Confidently facilitate Agile ceremonies when required
• Help refine and estimate upcoming work
• Raise technical risks and dependencies early
• Balance feature delivery with technical debt, operational support and platform improvements
• Work collaboratively with Product Owners, Delivery Managers, Architects and other engineering teams

Mindset

A successful Lead DevOps Engineer is someone who:

• Takes ownership rather than waiting to be directed
• Looks for ways to improve the platform, tooling and engineering practices
• Delivers solutions that are maintainable, secure and operationally sound
• Considers the long-term impact of technical decisions
• Shares knowledge freely and helps develop those around them
• Leads through technical expertise, collaboration and example
• Continuously learns and adapts to new technologies, engineering practices and platform requirements

• Building images
• Writing Dockerfiles
• Multi-stage builds
• Image optimisation
• Container troubleshooting

GitLab & CI/CD

All source code is hosted in GitLab.

CI/CD is implemented using GitLab CI.

Expected experience includes:

• Writing GitLab CI pipelines
• Maintaining reusable pipeline fragments and templates
• Building and maintaining reusable CI/CD components
• Managing multi-stage deployment pipelines across Development, Test, Stage and Production
• Pipeline troubleshooting
• Secure secret handling
• Build and deployment automation, * OAuth 2.0 flows
• OpenID Connect (OIDC)
• JWTs
• API authentication
• IAM
• Kubernetes RBAC

Knowledge of identity platforms and authentication patterns is highly desirable.

API Gateway

The platform makes extensive use of Kong API Gateway.

Useful experience includes:

• Routing
• Authentication plugins
• OIDC integration
• Rate limiting
• Upstream configuration
• Troubleshooting gateway behaviour

Service Mesh

• Takes ownership rather than waiting to be directed
• Looks for ways to improve the platform, tooling and engineering practices
• Delivers solutions that are maintainable, secure and operationally sound
• Considers the long-term impact of technical decisions
• Shares knowledge freely and helps develop those around them
• Leads through technical expertise, collaboration and example
• Continuously learns and adapts to new technologies, engineering practices and platform requirements