Software Engineer AI Evaluation & Automation : 26-02760

This role focuses on building and scaling the systems used to measure the performance of AI-powered software development tools. You will engineer evaluation frameworks and automate benchmarking processes to ensure reproducible and reliable results. The core of this position is ensuring the accuracy of measurements, not solely the automation of tasks., * Develop and integrate evaluation systems for AI development tools.

• Automate benchmark runs with versioned, reproducible environments.
• Calibrate evaluation metrics against human judgment for accuracy.
• Support performance benchmarking across quality and efficiency.
• Analyze and improve the reliability of evaluation workflows.

• Experience with AI coding tools and harnesses.
• Proven evaluation and benchmarking experience.
• Strong proficiency in Git and containerization with Docker.
• Python, Java or JavaScript

Industry Experience: Experience in software development, particularly with AI-assisted coding tools and their evaluation, is highly beneficial.

Akraya is an award-winning IT staffing firm consistently recognized for our commitment to excellence and a thriving work environment. Most recently, we were recognized Stevie Employer of the Year 2025, SIA Best Staffing Firm to work for 2025, Inc 5000 Best Workspaces in US (2025 & 2024) and Glassdoor’s Best Places to Work (2023 & 2022)!

Industry Leaders in Tech Staffing As Talent solutions provider for Fortune 100 Organizations, Akraya’s industry recognitions solidify our leadership position in the IT staffing space. We don’t just connect you with great jobs, we connect you with a workplace that inspires!

Join Akraya Today! Let us lead you to your dream career and experience the Akraya difference. Browse our open positions and join our team!

https://job.goodword.io/JobLanding/JobId=7c8fd0ed-c165-44de-aac3-42664d09b8ef?trafficsourceid=0 external