> Markdown version of [/jobs/ext/4629687-ai-evaluation-benchmark-engineer](https://www.wearedevelopers.com/jobs/ext/4629687-ai-evaluation-benchmark-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# AI Evaluation & Benchmark Engineer

- **Company:** THAMES, FREDERICK
- **Location:** San Francisco, CA, United States
- **Experience:** Experienced
- **Contract:** Permanent contract
- **Skills:** AI Evaluation, Application Programming Interfaces (APIs), Unit Testing, Continuous Integration, Python (Programming Language), Next.js, Tailwind, Evaluation of Large Language Models
- **Published:** November 7, 2026
- **Apply:** https://www.juju.com/job/28_25197_40ca17fd7a47693f7e0294596de63653

## About the Role

* 4+ years experience in Python, automated testing frameworks, and LLM evaluation tooling.
 * Strong knowledge of continuous integration (CI) and containerized execution.

## Description

* Build automated sandbox runners that execute and verify generated code across 10,000+ unit test suites.
 * Create synthetic benchmark datasets for Next.js, Tailwind, and full-stack API generation.
 * Identify model failure modes and partner with research teams on dataset filtering and RL feedback.