> Markdown version of [/jobs/ext/2241657-ai-evaluations-engineer](https://www.wearedevelopers.com/jobs/ext/2241657-ai-evaluations-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# AI Evaluations Engineer

- **Company:** ConnexAI
- **Location:** Manchester, UK
- **Contract:** Permanent contract
- **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Data Analysis, Automation of Tests, Python (Programming Language), Regression Testing, Scripting, Large Language Models, Machine Learning Operations
- **Published:** August 26, 2026
- **Apply:** https://www.collegerecruiter.com/job/2815157662-ai-evaluations-engineer

## About the Role

* You think in terms of system impact (before vs after), not just accuracy
 * You're comfortable working across code, data, and production systems
 * You care about real-world behaviour, not just benchmarks

Core skills

 * Strong Python (scripting, data analysis, tooling)
 * Experience with ML systems, evaluation, or experimentation
 * Understanding of LLMs or speech systems (ASR / TTS)
 * Ability to design test cases and structured datasets
 * Comfortable working with engineers and product teams, * Experience with LLM evaluation or benchmarking
 * Exposure to speech or multimodal systems
 * Familiarity with production APIs or ML systems
 * Experience with automated testing or CI-style workflows

## Description

This role sits at the centre of how we measure and improve AI systems in production.

You'll define what good performance means across LLMs, ASR, TTS, and full speech-to-speech pipelines, and build the datasets, metrics, and evaluation systems that make AI quality measurable and comparable in the real world.

You'll work closely with engineering and product teams to ensure model changes lead to real improvements in user experience, not just better offline benchmarks.

What you'll do

 * Design and run evaluations across LLM, ASR, TTS, and speech-to-speech systems
 * Build real-world datasets and test cases from production behaviour and edge cases
 * Define metrics and scorecards for model and system quality
 * Benchmark internal models against external and frontier systems
 * Build Python tools to automate evaluation workflows
 * Create internal leaderboards, red-teaming setups, and regression tests
 * Work with engineers and product teams to diagnose system failures
 * Turn vague product goals into measurable evaluation frameworks

What this role is about

 * Defining and measuring AI quality in production systems
 * Turning real user behaviour into structured evaluation signals
 * Ensuring model changes improve real-world performance
 * Understanding why AI systems fail, not just whether they do

What good looks like

 * You can translate improved quality into measurable metrics
 * You think in terms of system impact (before vs after), not just accuracy
 * You're comfortable working across code, data, and production systems
 * You care about real-world behaviour, not just benchmarks

Core skills

 * Strong Python (scripting, data analysis, tooling)
 * Experience with ML systems, evaluation, or experimentation
 * Understanding of LLMs or speech systems (ASR / TTS)
 * Ability to design test cases and structured datasets
 * Comfortable working with engineers and product teams

Nice to have

 * Experience with LLM evaluation or benchmarking
 * Exposure to speech or multimodal systems
 * Familiarity with production APIs or ML systems
 * Experience with automated testing or CI-style workflows


## Related Videos

- [JavaScript? No. Java Scripts! - Scripting with Java](https://www.wearedevelopers.com/videos/2094-javascript-no-java-scripts-scripting-with-java)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [Trunk-Based Development at Scale: Real-World Insights from a High-Traffic Luxury E-Commerce Platform](https://www.wearedevelopers.com/videos/1435-trunk-based-development-at-scale-real-world-insights-from-a-high-traffic-luxury-e-commerce-platform)
- [Three years of putting LLMs into Software - Lessons learned](https://www.wearedevelopers.com/videos/1508-three-years-of-putting-llms-into-software-lessons-learned)
- [Intermediate Bitcoin Script](https://www.wearedevelopers.com/videos/25-intermediate-bitcoin-script)
- [Evals vs. Evil - AI and Package Security - Laurie Voss](https://www.wearedevelopers.com/videos/2131-evals-vs-evil-ai-and-package-security-laurie-voss)

## Related Articles

- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)
- [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)
- [The State of WebDev AI 2025 Results: What Can We Learn?](https://www.wearedevelopers.com/magazine/581-the-state-of-webdev-ai-2025-results-what-can-we-learn)