AI Test Lead

Yorkshire Building Society
Leeds, UK
4 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Working hours
Regular working hours

Tech stack

Artificial Intelligence Automation of Tests Microsoft Azure DevOps Azure Machine Learning Delivery Pipeline Large Language Models Machine Learning Operations

Job description

Experteer Overview In this role you will establish and drive testing and quality assurance for AI-enabled systems, ensuring trustworthy and compliant deployment in a regulated financial services context. You will work with cross-functional teams to shape AI testing frameworks that handle probabilistic behavior and guardrails, while integrating automation into DevOps/MLOps. The position offers hybrid working with offices in Bradford and Leeds and values a people-centered, flexible culture. You will be instrumental in balancing innovation with risk and regulatory expectations to deliver reliable AI solutions. Pay / Benefits * Develop AI testing frameworks that account for nondeterminism, probabilistic outputs, and variability in AI responses * Embed responsible AI and risk testing by defining test cases for bias, fairness, transparency, safety and guardrails * Implement automated AI testing capabilities and integrate testing into DevOps and MLOps pipelines * Collaborate with delivery and governance teams to ensure testing aligns with engineering standards, regulatory expectations, and assurance requirements Tasks * Strong experience in testing and quality assurance for complex systems * Working knowledge of AI and machine learning technologies, including LLMs and RAG * Familiarity with test automation, DevOps pipelines, and MLOps practices * Experience with Microsoft Azure, Azure AI services, and Copilot-based solutions * Strong collaboration and communication skills with nontechnical stakeholders Key requirements * 25 days holiday plus Bank Holidays, with option to buy up to 5 additional days * flexible working options * on target bonus 7% of eligible pay (up to 15% maximum) * pension contributions up to 11% * private medical insurance and dental/healthcare plans * My Benefits portal with discounts

Requirements

_ governance teams to ensure testing aligns with engineering standards, regulatory expectations, and assurance requirements Tasks * Strong experience in testing and quality assurance for complex systems * Working knowledge of AI and machine learning technologies, including LLMs and RAG * Familiarity with test automation, DevOps pipelines, and MLOps practices * Experience with Microsoft Azure, Azure AI services, and Copilot-based solutions * Strong collaboration and communication skills with nontechnical stakeholders Key requirements * 25 days holiday plus Bank Holidays, with option to buy up to 5 additional days * flexible working options * on target bonus 7% of eligible pay (up to 15% maximum) * pension contributions up to 11% * private medical insurance and dental/healthcare plans * My Benefits portal with discounts

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on eu.experteer.com

Good distractions

Talks and stories from around this role — technically off-topic, practically not.

2:17 min

Mapping the maturity roadmap for scaled devops adoption

Dominik Krichbaum Dominik Krichbaum · World Congress 2026 Europe

1:06 min

Outline of free tools for Microsoft Azure

Radu Vunvulea Radu Vunvulea · World Congress 2022

1:51 min

Unifying software compliance into standard delivery pipelines

Marcus Ross Marcus Ross · World Congress 2026 Europe

2:06 min

Elevating the QA engineering role for complex challenges

Ondřej Gróf Ondřej Gróf · World Congress 2026 Europe

3:18 min

Scaling global network engineering through DevOps culture

Stuart Clark · LIVE

2:14 min

Exploring internal AI product initiatives and global engineering roles

Maria Apazoglou · Coffee With Developers

Videos

See all

Related articles

See all