> Markdown version of [/jobs/ext/4008729-software-engineer-ai-evaluation-automation-26-02760](https://www.wearedevelopers.com/jobs/ext/4008729-software-engineer-ai-evaluation-automation-26-02760). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Software Engineer AI Evaluation & Automation : 26-02760

- **Company:** Akraya, Inc.
- **Location:** San Diego, CA, United States (Remote available)
- **Salary:** $135,200.0 - $151,840.0
- **Contract:** Temporary to permanent
- **Skills:** AI Evaluation, Java (Programming Language), JavaScript (Programming Language), Artificial Intelligence, Python (Programming Language), Software Engineering, AI Coding Agents, Git, Containerization, Docker
- **Published:** October 29, 2026
- **Apply:** https://www.thejobnetwork.com/job/7c8fd0ed-c165-44de-aac3-42664d09b8ef/software-engineer-ai-evaluation-automation-

## About the Role

* Experience with AI coding tools and harnesses.
 * Proven evaluation and benchmarking experience.
 * Strong proficiency in Git and containerization with Docker.
 * Python, Java or JavaScript

Industry Experience:
Experience in software development, particularly with AI-assisted coding tools and their evaluation, is highly beneficial.

## Description

This role focuses on building and scaling the systems used to measure the performance of AI-powered software development tools. You will engineer evaluation frameworks and automate benchmarking processes to ensure reproducible and reliable results. The core of this position is ensuring the accuracy of measurements, not solely the automation of tasks., * Develop and integrate evaluation systems for AI development tools.

 * Automate benchmark runs with versioned, reproducible environments.
 * Calibrate evaluation metrics against human judgment for accuracy.
 * Support performance benchmarking across quality and efficiency.
 * Analyze and improve the reliability of evaluation workflows.