> Markdown version of [/jobs/ext/2875452-software-engineer-ai-evaluation-automation](https://www.wearedevelopers.com/jobs/ext/2875452-software-engineer-ai-evaluation-automation). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Software Engineer - AI Evaluation & Automation

- **Company:** IDR, Inc.
- **Location:** Santa Clara, CA, United States (Remote available)
- **Contract:** Permanent contract
- **Skills:** Java (Programming Language), JavaScript (Programming Language), Application Programming Interfaces (APIs), Artificial Intelligence, Programming Tools, Python (Programming Language), Software Engineering, Large Language Models, Git, Containerization, Docker
- **Published:** September 13, 2026
- **Apply:** https://www.indeed.com/viewjob?jk=ba5c5fc7cefb8684

## About the Role

* Strong software engineering background, with experience building automation, developer tooling, or validation systems.
 * Proficient in Python, and comfortable with Java, JavaScript, or similar languages.
 * Solid working knowledge of Git, including branch management, history, and worktrees, and containerization with Docker.
 * Experience with APIs, development environments, CI/CD pipelines, and engineering workflows.
 * Some familiarity with AI, LLM, or agent evaluation processes, including understanding common pitfalls.

## Description

* Help build and scale tooling used to measure the performance of AI-powered software development tools.
 * Develop evaluation harnesses, automate benchmark runs, and ensure results are reproducible and trustworthy.
 * Focus on validating measurement approaches against human judgment and analyzing benchmarking results.
 * Collaborate with engineering and data teams to improve tooling and document evaluation processes.
 * Support execution-based benchmarking across quality, productivity, and efficiency measures, including cost and latency.


## Related Videos

- [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered)
- [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94)
- [Containers and Kubernetes made easy: Deep dive into Podman Desktop and new AI capabilities](https://www.wearedevelopers.com/videos/1133-containers-and-kubernetes-made-easy-deep-dive-into-podman-desktop-and-new-ai-capabilities)
- [Developer Experience in the Age of AI](https://www.wearedevelopers.com/videos/1118-developer-experience-in-the-age-of-ai)
- [Turning Container security up to 11 with Capabilities](https://www.wearedevelopers.com/videos/718-turning-container-security-up-to-11-with-capabilities)
- [Git for Code Reviews](https://www.wearedevelopers.com/videos/429-git-for-code-reviews)

## Related Articles

- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [The State of WebDev AI 2025 Results: What Can We Learn?](https://www.wearedevelopers.com/magazine/581-the-state-of-webdev-ai-2025-results-what-can-we-learn)
- [Dev Digest 137 - AI'm not sure about this](https://www.wearedevelopers.com/magazine/485-dev-digest-137-ai-m-not-sure-about-this)
- [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline)
- [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)