> Markdown version of [/jobs/ext/2951357-ai-evaluation-engineer](https://www.wearedevelopers.com/jobs/ext/2951357-ai-evaluation-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# AI Evaluation Engineer

- **Company:** SmartSourcing Ltd
- **Location:** London, UK
- **Salary:** £182,000.0 - £195,000.0
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Application Layers, Automation of Tests, Python (Programming Language), Machine Learning, Large Language Models, SC Clearance, Data Analytics, Codebase, SDET
- **Published:** September 17, 2026
- **Apply:** https://find.jobs/jobs-near-me/apply/ats-redirect/?id=2974755480-2

## About the Role

Security Clearance: Active SC Clearance preferred / eligible and willing to undergo SC Clearance.

Alternative backgrounds may include SDET, AI Evaluation Engineer, AI Assurance Engineer, Prompt Engineer, Harness Engineer or AI Quality Engineer., Strong commercial experience working with AI, Machine Learning and LLM-based systems in production environments.
Deep understanding of AI evaluation methodologies, frameworks and quality engineering principles.
Hands-on Python development experience with the ability to create and maintain your own codebase.
Experience evaluating AI solutions across:
Model layer
Agentic layer
Application layer
Knowledge of AI evaluation tooling and frameworks such as:
RAGAS
DeepEval (DP-Eval)
Other AI evaluation and testing frameworks
Experience developing automated testing, evaluation harnesses and assurance processes for AI systems.
Proven leadership experience, including working across teams, departments and organisational boundaries.
Strong stakeholder engagement and communication skills.

Desirable Experience
Public sector or government programme experience.
Understanding of responsible AI and AI assurance frameworks.

## Description

This is a hands-on engineering role for someone with deep expertise in AI/ML evaluation, testing and quality assurance, rather than a cloud, infrastructure or platform engineer. You will play a key role in establishing robust evaluation frameworks for production-grade AI systems, helping assure the quality, safety and effectiveness of data-driven and AI-enabled products., Design, develop and deploy AI evaluation and risk management tooling.
Build and maintain evaluation frameworks for AI, ML and LLM-based products.
Create and develop code solutions, harnesses and testing capabilities from the ground up using Python.
Define and execute evaluation strategies across model, agentic and application layers.
Assess AI system quality, performance, reliability, safety and risk.
Work closely with technical and non-technical stakeholders across multiple departments and organisations.
Provide technical leadership and guidance on AI evaluation best practice.


## Related Videos

- [Getting to Know Your Legacy (System) with AI-Driven Software Archeology](https://www.wearedevelopers.com/videos/1437-getting-to-know-your-legacy-system-with-ai-driven-software-archeology)
- [Introduction to Responsible AI: Balancing Value and Risk](https://www.wearedevelopers.com/videos/1972-introduction-to-responsible-ai-balancing-value-and-risk)
- [Why your codebase lies to AI?](https://www.wearedevelopers.com/videos/100281-why-your-codebase-lies-to-ai)
- [Testing AI Agents: Automated Evaluation for Chatbots & RAG Systems](https://www.wearedevelopers.com/videos/100300-testing-ai-agents-automated-evaluation-for-chatbots-rag-systems)
- [Evals vs. Evil - AI and Package Security - Laurie Voss](https://www.wearedevelopers.com/videos/2131-evals-vs-evil-ai-and-package-security-laurie-voss)
- [Refactoring in the Age of AI](https://www.wearedevelopers.com/videos/100223-refactoring-in-the-age-of-ai)

## Related Articles

- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [What Industries Outside of AI Are Hiring The Most AI Experts?](https://www.wearedevelopers.com/magazine/98-what-industries-outside-of-ai-are-hiring-the-most-ai-experts)
- [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)
- [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud)
- [Trustworthy AI Starts at Deployment: 5 Checks Before You Ship](https://www.wearedevelopers.com/magazine/753-trustworthy-ai-starts-at-deployment-5-checks-before-you-ship)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)