> Markdown version of [/jobs/ext/4207653-research-engineer-data-and-evaluation](https://www.wearedevelopers.com/jobs/ext/4207653-research-engineer-data-and-evaluation). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Research Engineer - Data and Evaluation

- **Company:** Prime Inc.
- **Location:** San Francisco, CA, United States (Remote available)
- **Salary:** $150,000.0 - $350,000.0
- **Contract:** Permanent contract
- **Skills:** Training Data, Software Debugging, Distributed Computing Environment, Python (Programming Language), Language Modeling, Software Engineering, AI Coding Agents, Build Management, Machine Learning Operations, Data Pipelines, Data Generation
- **Published:** October 15, 2026
- **Apply:** https://www.indeed.com/viewjob?jk=6181d8805a23855d

## About the Role

* Strong software engineering skills in Python and experience building data pipelines, evaluation systems, or ML workflows used by others.
 * Hands-on experience working with language models or agents, with an understanding of how training data and evaluation choices affect model behavior.
 * Ability to inspect complex data and model outputs, identify failure modes, and turn an ambiguous quality question into a testable measurement.
 * A record of owning technical projects through experimentation, implementation, and validation, while collaborating closely with researchers and engineers.
 * Hands-on experience building technical evaluations or agent environments, with the engineering depth to inspect model traces and debug harness and scoring code.

Relevant experience may include:

 * Large-scale synthetic data generation, dataset curation, or distributed processing for model training.
 * Evaluation methodology, executable coding tasks, agent environments, verifiers, or reward design.
 * Experiment analysis, statistical testing, contamination detection, and reproducible research tooling.

Nice to have

 * Experience with RL or post-training data, including task difficulty selection and analysis of rollout or reward distributions.
 * Contributions to Verifiers, Prime-RL, or comparable environment and evaluation frameworks.
 * Work on model-based grading, long-horizon coding agents, or diagnosing reward exploitation.

We welcome depth in data generation, environments and rewards, or evaluation systems. You don't need to have worked across every area.

## Description

* Design and build challenging evaluations for coding and long-horizon agents.
 * Build executable environments in Verifiers, with realistic task setup and clear success criteria.
 * Develop and test scoring logic and verifiers, investigating reward hacks and misleading measures of progress.
 * Trace failures through model outputs and harness code to distinguish model limitations from problems in the environment or scoring.
 * Build and scale RL data pipelines that generate and curate challenging coding and agentic tasks.
 * Work with RL researchers to test whether changes to tasks and training data improve performance on held-out tasks.