AI QA Tester

Lorien

London, UK

13 days ago

Apply on www.careerboard.com

Prepare application

Role details

Contract type

Temporary contract

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Job source

www.careerboard.com

Tech stack

JavaScript (Programming Language) Application Programming Interfaces (APIs) Artificial Intelligence Application Performance Management Automation of Tests Microsoft Azure C Sharp (Programming Language) Continuous Integration Document-Oriented Databases Github Apache JMeter Python (Programming Language)

+15 more

Load Testing Machine Learning Open Web Application Security Kusto Query Language Strategies of Testing TypeScript Scripting Performance Testing Postman Large Language Models Concurrency Pytest Playwright Data Management Api Management

Job description

Own and maintain the test strategy for the agentic platform, spanning the agent orchestration layer, the data APIs, the retrieval pipeline and the underlying Azure services.
Build and curate golden test datasets and expected-outcome sets with input from business subject-matter experts, covering happy paths, edge cases, ambiguous questions and out-of-scope requests.
Design and automate LLM evaluation: groundedness/faithfulness, answer relevance, retrieval precision and recall, citation correctness, task completion, tone and consistency - using Azure AI Foundry evaluation, LLM-as-judge and deterministic checks as appropriate.
Wire evaluations into CI/CD as release gates, so prompt, model, index or code changes cannot regress quality unnoticed; report trends over time.
Carry out adversarial and red-team testing: prompt injection, jailbreaks, indirect injection via knowledge-base content, data-exfiltration attempts, tool misuse and privilege escalation through agent tool calls.
Validate the safety and monitoring controls - Azure AI Content Safety categories and thresholds, refusal and escalation behaviour, and Azure AI Anomaly Detector signals - including deliberate false-negative and false-positive probing.
Test authorisation rigorously: confirm that agents and APIs never return member or document data outside the requesting user’s entitlement, including via retrieval or summarisation side channels.
Build automated functional and contract test suites for the Agent Data API and Member Data API (for example pytest, Playwright, Postman/Newman, RestSharp) and for the end-to-end conversational flows.
Run non-functional testing: latency and response-time budgets, throughput and concurrency, token cost per interaction, rate-limit and failover behaviour, and resilience when BCUK or a dependency degrades.

Requirements

Strong QA engineering background with hands-on test automation, not manual Scripting alone.
Automation coding ability in Python, C# and/or JavaScript/TypeScript.
API testing experience, including contract testing, authentication, negative testing and mocked dependencies.
Demonstrable experience testing AI, machine learning or LLM-based systems, or a clear grasp of how to assure non-deterministic output using evaluation metrics and statistical thresholds.
Understanding of RAG architectures and where they fail - retrieval misses, stale indexes, chunk boundary loss, hallucination, ungrounded citations.
Awareness of LLM security risks (for example the OWASP Top 10 for LLM applications) and practical adversarial testing technique.
CI/CD integration of automated test suites (GitHub Actions or Azure DevOps).
Test data management, including handling of personal data safely in test environments.
Confidence challenging engineers and stakeholders on release readiness.
Experience with Azure AI Foundry evaluations, Azure AI Content Safety testing or PyRIT/other red-teaming tooling.
Performance and load testing tools (k6, JMeter, Azure Load Testing).
Azure Monitor/Application Insights and KQL for investigating failures from telemetry.
Accessibility and conversational UX testing experience.
Familiarity with UK GDPR, DPIA processes or regulated-sector assurance evidence.