AI Test Automation Engineer - FTC until the end of the year - Hybrid/London
Robson Bale Ltd
Charing Cross, United Kingdom
6 days ago
Role details
Contract type
Temporary contract Employment type
Full-time (> 32 hours) Working hours
Regular working hours Languages
EnglishJob location
Charing Cross, United Kingdom
Tech stack
API
Artificial Intelligence
Automation of Tests
Cloud Computing
Continuous Integration
Distributed Systems
Python
Strategies of Testing
Data Logging
Large Language Models
Pytest
Integration Tests
Cucumber
Network Server
Automation Anywhere
Microservices
Job description
We are looking for an AI Test Automation Engineer to join on a fixed-term contract until the end of the year, with potential to extend. The role is based in London, with some office presence required.
You will help design and implement automated testing and evaluation frameworks for MCP Servers, LLM-based systems, and agentic AI workflows, ensuring accuracy, safety, reliability, and performance across complex AI-enabled platforms.
Key Responsibilities
- Develop automated testing frameworks for MCP Servers and related AI systems.
- Design evaluation strategies for LLM accuracy, safety, and reliability.
- Build automated tests using Python, Pytest, and BDD frameworks.
- Integrate quality gates into CI/CD pipelines.
- Identify and address AI failure modes including hallucination, latency, and incorrect tool usage.
- Work with engineering, QA, and product teams to define quality metrics and acceptance criteria.
- Support Agile delivery, ensuring testing aligns with sprint goals.
- Produce reporting on quality outcomes, risks, and improvement areas.
- Maintain documentation for test cases, evaluation pipelines, and validation strategies.
Requirements
- Strong Python programming experience for test automation and evaluation.
- Expertise in Pytest and familiarity with BDD tools such as Behave or Cucumber.
- Knowledge of LLM evaluation approaches such as RAGAS, DeepEval, or custom evaluation pipelines.
- Understanding of agentic AI issues including hallucination, tool misuse, and performance bottlenecks.
- Experience testing AI workflows, distributed systems, or microservices environments.
- Strong knowledge of Agile delivery and CI/CD quality integration.
- Excellent communication, analytical, and problem-solving skills.
Nice to Have
- Experience with Model Context Protocol (MCP) or agent orchestration solutions.
- Exposure to observability, monitoring, or logging tools for AI systems.
- Background in API and service integration testing.
- Knowledge of containerised and cloud-native environments.
- Experience in enterprise AI automation or intelligent platform engineering.