> Markdown version of [/jobs/ext/4806109-software-engineer-ai-evaluation-automation-26-03500](https://www.wearedevelopers.com/jobs/ext/4806109-software-engineer-ai-evaluation-automation-26-03500). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Software Engineer AI Evaluation & Automation : 26-03500 - **Company:** Akraya, Inc. - **Location:** San Diego, CA, United States (Remote available) - **Salary:** $124,800 - $133,120 - **Contract:** Temporary to permanent - **Skills:** AI Evaluation, Artificial Intelligence, Programming Tools, Python (Programming Language), Performance Tuning, Software Engineering, Scripting, Performance Testing, AI Coding Agents, Git, Docker - **Published:** November 20, 2026 - **Apply:** https://www.careerbuilder.com/job-details/software-engineer-ai-evaluation-automation-26-03500-san-diego-ca--378a9f25-3997-4f5f-b6c1-f6a9b4117c67 ## About the Role Software Engineering, Artificial Intelligence (AI), Automation, Software Evaluation, Artificial Intelligence (AI), Python Programming/Scripting Language, Docker, Git, Programming Tools, Software Development, * Proven experience using AI coding tools and agentic harnesses. * Hands-on experience with Git and Docker for reproducibility. * Expertise in building automation and developer tooling systems. Industry Experience: This role requires experience in software engineering, particularly with automation and validation systems, and an understanding of how to evaluate AI performance effectively. ## Description This role focuses on building and scaling the systems used to measure the effectiveness of AI-powered software development tools. You will develop evaluation frameworks, automate performance tests, and ensure the accuracy and reproducibility of results. The position emphasizes rigorous measurement and validation, not just automation., * Develop and integrate evaluation harnesses for software development tasks. * Establish versioned, repeatable processes for evaluating AI tools and models. * Validate evaluation approaches against human judgment for score accuracy. * Support execution-based benchmarking across quality, productivity, and efficiency metrics. * Analyze test results to identify variance, failure patterns, and cost per outcome.