> Markdown version of [/jobs/ext/1321768-machine-learning-engineer-evals](https://www.wearedevelopers.com/jobs/ext/1321768-machine-learning-engineer-evals). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Machine Learning Engineer, Evals

- **Company:** Nous Research, Inc.
- **Location:** United States (Remote available)
- **Experience:** Experienced
- **Contract:** Internship / Graduate position
- **Skills:** Training Data, Continuous Integration, Software Debugging, Python (Programming Language), Linux Commands, Machine Learning, Open Source Technology, Software Engineering, Large Language Models, Git, Docker
- **Published:** July 17, 2026
- **Apply:** https://www.indeed.com/viewjob?jk=66f0e0a55de940a7

## About the Role

* 3+ years in software engineering, ML engineering, data science, or a research-adjacent role, with concrete evaluation experience from coursework, an internship, a side project, open source work, or a job
 * Experience with at least one LLM evaluation framework (Harbor, Nemo Evaluator, etc.), with real opinions on what it does well and where it falls short
 * Hands-on experience with LLMs: prompting, few-shot design, and ideally fine-tuning or RAG; regular use of coding agents
 * Solid Python. You write clean, tested, version-controlled code that a colleague could run without you babysitting it
 * Comfort with Git, CI/CD basics, Docker, and the Linux command line (SSH, tmux, debugging a remote job)
 * Understanding of basic eval statistics: why accuracy misleads on imbalanced judges, what Cohen's κ measures, how to think about confidence intervals on a metric
 * At least 3 of the following: you can explain why LLM-as-judge needs calibration; you've done failure analysis and can tell model bugs apart from prompt, grader, or retrieval issues; you know at least two agent benchmarks (GAIA, AgentBench, τ-Bench, MINT, SWE-bench, WebShop, ALFWorld) and a limitation of each; you've designed or extended an eval dataset with happy paths, edge cases, and adversarial examples; you've thought about non-determinism in eval, how you sample, how many runs, how you report variance
 * You communicate clearly to both researchers and engineers, in the right language for each
 * You're comfortable with ambiguity, can turn a half-formed request into a plan, and know when to ask for help

Preferred

 * RLVR / RLHF pipeline experience
 * Training data curation experience
 * Distributed eval orchestration experience
 * Benchmark design from scratch
 * Red teaming and adversarial eval experience
 * Familiarity with psychometrics or measurement theory

## Description

You'll work across the lab on agent capability evals, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties it together. This is a high-growth, high-ownership role on a small team, and you'll ship evaluation infrastructure that researchers depend on from day one., * Run the full eval pipeline end to end and reproduce known results during onboarding, pairing with a senior engineer on your first task

 * Build a judge calibration protocol: sample human-labeled decisions, measure agreement (κ, per-class P/R), identify drift zones, and document it so anyone can re-run it
 * Extend an existing benchmark (GAIA, τ-Bench, SWE-bench slice, etc.) with new tasks targeting known capability gaps, including the prompt, environment, rubric, automated grader, and QA
 * Run failure analysis on model outputs: categorize failure modes, quantify prevalence, and write up findings with recommendations for training data, judge prompts, or benchmark changes
 * Own a recurring eval workflow (weekly regression suite, judge drift dashboard, red-team evaluation for a new capability) and ship tooling researchers actually use


## Related Videos

- [Evals vs. Evil - AI and Package Security - Laurie Voss](https://www.wearedevelopers.com/videos/2131-evals-vs-evil-ai-and-package-security-laurie-voss)
- [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94)
- [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered)
- [Fireside Chat: Deep Learning, Deep Impact: Harnessing AI for Language Innovation](https://www.wearedevelopers.com/videos/612-fireside-chat-deep-learning-deep-impact-harnessing-ai-for-language-innovation)
- [Why LLMs Need Observability and How to Do It](https://www.wearedevelopers.com/videos/2117-why-llms-need-observability-and-how-to-do-it)
- [Git for Code Reviews](https://www.wearedevelopers.com/videos/429-git-for-code-reviews)

## Related Articles

- [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline)
- [Dev Digest 137 - AI'm not sure about this](https://www.wearedevelopers.com/magazine/485-dev-digest-137-ai-m-not-sure-about-this)
- [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)