> Markdown version of [/jobs/ext/3051329-software-engineering-director-agentic-evaluations](https://www.wearedevelopers.com/jobs/ext/3051329-software-engineering-director-agentic-evaluations). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Software Engineering Director, Agentic Evaluations - **Company:** G2 LLC - **Location:** United States (Remote available) - **Experience:** Expert - **Salary:** $260,000.0 - $320,000.0 - **Contract:** Permanent contract - **Skills:** Java (Programming Language), JavaScript (Programming Language), Artificial Intelligence, Amazon Web Services, Computer Programming, Python (Programming Language), Node.Js, Next.js, TypeScript, Large Language Models, Backend, Fastapi, Kotlin, Playwright, Cloudflare - **Published:** September 24, 2026 - **Apply:** https://www.builtincolorado.com/job/software-engineering-director-agentic-evaluations/11337453?handler=ApplyRedirect ## About the Role We realize applying for jobs can feel daunting at times. Even if you don't check all the boxes in the job description, we encourage you to apply anyway. * 10+ years of professional programming experience in backend or full-stack environments * 2+ years of experience directly managing engineers * Expert-level proficiency in backend development using languages like Python, Java/Kotlin, Typescript/Javascript or Go; strong proficiency in associated backend frameworks like FastAPI or Node.js * Direct experience creating evaluations (evals) against a customer-facing agent, leveraging agent trajectory trace data and rubrics to measure an agent's task completion rate, accuracy, correctness, or policy adherence * Direct experience using frontier models from OpenAI, Anthropic, or Google in LLM-as-a-judge applications * Regular use of coding agent harnesses like Claude Code, Codex, Opencode, or Pi as part of the daily development workflow * BS/BA degree in related field What Can Help Your Application Stand Out: * Experience with agent tool use via direct integration or as mediated by MCP servers * Knowledge of STATE-Bench, tau2-bench or similar benchmark frameworks * Experience using Playwright, browser-use, Chrome DevTools MCP or others to drive user flows in the browser * Experience with durable execution frameworks & workflow solutions, like Temporal, DBOS, Cloudflare / Vercel Workflows, or similar; or experience with agent sandboxing using AWS E2B, Daytona, Cloudflare / Vercel Containers or similar, Leads business development and capture activities for Latin American defense and government customers. Identifies opportunities, analyzes customer and competitor needs, builds senior stakeholder relationships, coordinates bids and proposals, shapes requirements, manages campaign plans, and supports trade shows and customer engagements. Requires extensive aerospace or defense experience, regional customer expertise, cross-cultural fluency, knowledge of U.S. and international defense sales frameworks, export controls, local partnerships, and English proficiency; Spanish and Portuguese are preferred. Top Skills: CRMDirect Commercial Sales (Dcs)EarForeign Military Sales (Fms)ItarPipeline Management ToolsUncrewed Air Systems ## Description As a Software Engineering Director, Agentic Evaluations, your responsibilities will center around delivery of useful and credible evaluations of agents offered by software vendors. You'll enhance the system that performs validation of agent performance, and lead the core team behind it. You will leverage "agent-first" ways of working, applying agentic engineering techniques to deliver your work to production. Detailed Responsibilities: Build and enhance agent evaluations that run against live agents [80%]: * Help scope out the feasibility and effort involved in evaluating an agent offered by a software vendor, identifying the most practical path to reaching a credible evaluation * Own the full stack of the core evaluation system, from admin & API surface to eval workflow dispatch * Design evaluation system primitives that generalize across software verticals, so onboarding a new category benefits from reuse * Distill the appropriate and repeatable processes in onboarding and maintaining integrations into repeatable AI skills or agents to increase evaluation velocity * Keep tabs on emerging frameworks and techniques for agent evaluation which offer improvements to evaluations, and provide industry-accepted means to distribute eval results * Collaborate with data science peers in their pursuit to develop rich and proprietary benchmarks Lead a group of engineers and be a source of wider influence [20%]: * You will facilitate the growth and mentorship of a core team of engineers, and be a source of technical leadership and subject matter expertise in evals for them * You will also help socialize the use of evals in agent-oriented products built by the wider organization. ## Related Videos - [Building a Multi-Agent Orchestration Engine That Actually Follows the Rules](https://www.wearedevelopers.com/videos/100159-building-a-multi-agent-orchestration-engine-that-actually-follows-the-rules) - [Kotlin Multiplatform - True power of native code reuse](https://www.wearedevelopers.com/videos/4-kotlin-multiplatform-true-power-of-native-code-reuse) - [Stop using Node.js like in 2020! What changed and what you can do today with Node.js](https://www.wearedevelopers.com/videos/100011-stop-using-node-js-like-in-2020-what-changed-and-what-you-can-do-today-with-node-js) - [GraphQL + Apollo + Next.js: A Lovely Trio](https://www.wearedevelopers.com/videos/311-graphql-apollo-next-js-a-lovely-trio) - [Building Agents Securely at Scale - Alfonso Graziano](https://www.wearedevelopers.com/videos/1862-building-agents-securely-at-scale-alfonso-graziano) - [Why Kotlin is the better Java and how you can start using it](https://www.wearedevelopers.com/videos/661-why-kotlin-is-the-better-java-and-how-you-can-start-using-it) ## Related Articles - [What is Agentic Programming and Why Should Developers Care?](https://www.wearedevelopers.com/magazine/625-what-is-agentic-programming-and-why-should-developers-care) - [Is Software Engineering Over-Saturated?](https://www.wearedevelopers.com/magazine/418-is-software-engineering-over-saturated) - [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline) - [Fully Remote Software Engineer Jobs](https://www.wearedevelopers.com/magazine/447-fully-remote-software-engineer-jobs) - [The Prompt Engineer ✍️](https://www.wearedevelopers.com/magazine/216-the-prompt-engineer) - [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)