> Markdown version of [/jobs/ext/4773770-evaluations-software-engineer](https://www.wearedevelopers.com/jobs/ext/4773770-evaluations-software-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Evaluations Software Engineer - **Company:** XDOF INC. - **Location:** San Mateo, United States - **Contract:** Permanent contract - **Skills:** C++ (Programming Language), Computer Programming, Python (Programming Language), Machine Learning, Open Source Technology, Evaluation Pipelines, Large Language Models, AI Coding Agents, Isaac Sim - **Published:** November 18, 2026 - **Apply:** https://startup.jobs/member-of-technical-staff-evaluations-xdof-10812484 ## About the Role * strong programming skills in Python or C++ * built benchmarks, evaluation pipelines, or graders for ML models, and thought hard about what a score actually measures * hands-on experience with robot learning, simulation (MuJoCo, Isaac Sim, or similar), or deploying policies on real hardware * worked with LLM or coding agents, and can read a trace to diagnose why a model failed * an adversarial mindset: your first question about a metric is how it could be gamed * built and shipped software in fast-moving environments Bonus points if you have: * contributed to robotics or agent benchmarks (e.g. LIBERO, BEHAVIOR, RoboArena, Terminal-Bench) * experience with RL, reward modeling, or post-training * worked on real2sim, system identification, or sim-to-real transfer * experience with statistics for evaluation, like confidence intervals and variance across runs * familiarity with containerized evaluation frameworks such as Harbor ## Description Evaluations engineers own how we measure models and what we learn from it. Sample projects include: * designing benchmark tasks for LLMs, VLAs, and WAMs that test real robotics capabilities, from long-horizon planning to dexterous manipulation * building verifiers, success detectors, and progress metrics that grade episodes reliably in simulation and on real hardware * building harnesses that let LLMs and coding agents control robots, directly or by planning over VLAs * measuring how well simulation results predict real-world performance, and improving that correlation * fine-tuning open-source VLAs consistently so every model gets a fair baseline * analyzing failures across runs to find capability gaps, and turning them into targeted data collection and RL environments * producing partner-facing results with rigorous statistics and clear reporting