Software Engineer AI Evaluation & Automation : 26-03500
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
Job description
This role focuses on building and scaling the systems used to measure the effectiveness of AI-powered software development tools. You will develop evaluation frameworks, automate performance tests, and ensure the accuracy and reproducibility of results. The position emphasizes rigorous measurement and validation, not just automation., * Develop and integrate evaluation harnesses for software development tasks.
- Establish versioned, repeatable processes for evaluating AI tools and models.
- Validate evaluation approaches against human judgment for score accuracy.
- Support execution-based benchmarking across quality, productivity, and efficiency metrics.
- Analyze test results to identify variance, failure patterns, and cost per outcome.
Requirements
Software Engineering, Artificial Intelligence (AI), Automation, Software Evaluation, Artificial Intelligence (AI), Python Programming/Scripting Language, Docker, Git, Programming Tools, Software Development, * Proven experience using AI coding tools and agentic harnesses.
- Hands-on experience with Git and Docker for reproducibility.
- Expertise in building automation and developer tooling systems.
Industry Experience: This role requires experience in software engineering, particularly with automation and validation systems, and an understanding of how to evaluate AI performance effectively.
About the company
Akraya is an award-winning IT staffing firm consistently recognized for our commitment to excellence and a thriving work environment. Most recently, we were recognized Stevie Employer of the Year 2025, SIA Best Staffing Firm to work for 2025, Inc 5000 Best Workspaces in US (2025 & 2024) and Glassdoor’s Best Places to Work (2023 & 2022)!
Industry Leaders in Tech Staffing As Talent solutions provider for Fortune 100 Organizations, Akraya’s industry recognitions solidify our leadership position in the IT staffing space. We don’t just connect you with great jobs, we connect you with a workplace that inspires!
Join Akraya Today! Let us lead you to your dream career and experience the Akraya difference. Browse our open positions and join our team!
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this role…