> Markdown version of [/jobs/ext/4774633-applied-ai-researcher-benchmarking](https://www.wearedevelopers.com/jobs/ext/4774633-applied-ai-researcher-benchmarking). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Applied AI Researcher, Benchmarking - **Company:** Distyl AI - **Location:** San Francisco, CA, United States - **Salary:** $150,000 - $250,000 - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Data Analysis, Computer Programming, Cursor, Perplexity AI, ReactJS, GPT - **Published:** November 18, 2026 - **Apply:** https://www.careerbuilder.com/job-details/applied-ai-researcher-benchmarking-san-francisco-ca--b7dea83d-386b-41f9-8149-85b3a3f04711 ## About the Role * Experience Designing and Running Evaluations: You've built or maintained benchmarks, test suites, or experimental frameworks to measure model or system performance * Statistical and Analytical Rigor: You design fair, reproducible experiments and can extract signal from noisy empirical results * Experience Building with Models, Not Just Building Models: We develop intelligent systems using models rather than training or fine-tuning them. Ideal candidates have expertise in compound AI systems, agentic collaboration, and associated techniques (ensembling, ReAct, graph-of-thoughts, etc.) * Proven Track Record of Research Results: Whether you've published in top journals, posted amazing work on twitter, or somewhere else we want to see what you've done * Uses AI Every Day: Before you can revolutionize someone else's workflow, you need to revolutionize yours. You should be using tools like ChatGPT, Cursor, and Perplexity to accelerate your workflow * Strong Programming and Data Analysis Skills: While you might not consider yourself a software engineer you need to be able to build prototypes of your ideas and then perform the experiments to prove the effectiveness to a F500 Head of AI * Biases Towards Showing vs Telling: Our customers want to see the power of AI today vs discuss the most elegant idea that will take 5 years to realize ## Description * The Benchmarking team defines how progress is measured. Researchers design evaluation frameworks that capture reasoning depth, interaction quality, reliability, and operational impact. They construct benchmarks that reflect real-world complexity. Their systems become the standard by which new architectures, techniques, and releases are judged. * Researchers in Benchmarking explore new paradigms for evaluating intelligent systems: adversarial robustness testing, longitudinal performance tracking, and human-in-the-loop assessment. They investigate how metrics shape model behavior and establish rigorous methodologies for quantifying emergent capability. Their insights drive both Distyl's internal research priorities and industry-wide standards.