Software Engineering Director, Agentic Evaluations

As a Software Engineering Director, Agentic Evaluations, your responsibilities will center around delivery of useful and credible evaluations of agents offered by software vendors. You’ll enhance the system that performs validation of agent performance, and lead the core team behind it. You will leverage “agent-first” ways of working, applying agentic engineering techniques to deliver your work to production.

Detailed Responsibilities:

Build and enhance agent evaluations that run against live agents [80%]:

• Help scope out the feasibility and effort involved in evaluating an agent offered by a software vendor, identifying the most practical path to reaching a credible evaluation
• Own the full stack of the core evaluation system, from admin & API surface to eval workflow dispatch
• Design evaluation system primitives that generalize across software verticals, so onboarding a new category benefits from reuse
• Distill the appropriate and repeatable processes in onboarding and maintaining integrations into repeatable AI skills or agents to increase evaluation velocity
• Keep tabs on emerging frameworks and techniques for agent evaluation which offer improvements to evaluations, and provide industry-accepted means to distribute eval results
• Collaborate with data science peers in their pursuit to develop rich and proprietary benchmarks

Lead a group of engineers and be a source of wider influence [20%]:

• You will facilitate the growth and mentorship of a core team of engineers, and be a source of technical leadership and subject matter expertise in evals for them
• You will also help socialize the use of evals in agent-oriented products built by the wider organization.

We realize applying for jobs can feel daunting at times. Even if you don’t check all the boxes in the job description, we encourage you to apply anyway.

• 10+ years of professional programming experience in backend or full-stack environments
• 2+ years of experience directly managing engineers
• Expert-level proficiency in backend development using languages like Python, Java/Kotlin, Typescript/Javascript or Go; strong proficiency in associated backend frameworks like FastAPI or Node.js
• Direct experience creating evaluations (evals) against a customer-facing agent, leveraging agent trajectory trace data and rubrics to measure an agent’s task completion rate, accuracy, correctness, or policy adherence
• Direct experience using frontier models from OpenAI, Anthropic, or Google in LLM-as-a-judge applications
• Regular use of coding agent harnesses like Claude Code, Codex, Opencode, or Pi as part of the daily development workflow
• BS/BA degree in related field

What Can Help Your Application Stand Out:

• Experience with agent tool use via direct integration or as mediated by MCP servers
• Knowledge of STATE-Bench, tau2-bench or similar benchmark frameworks
• Experience using Playwright, browser-use, Chrome DevTools MCP or others to drive user flows in the browser
• Experience with durable execution frameworks & workflow solutions, like Temporal, DBOS, Cloudflare / Vercel Workflows, or similar; or experience with agent sandboxing using AWS E2B, Daytona, Cloudflare / Vercel Containers or similar, Leads business development and capture activities for Latin American defense and government customers. Identifies opportunities, analyzes customer and competitor needs, builds senior stakeholder relationships, coordinates bids and proposals, shapes requirements, manages campaign plans, and supports trade shows and customer engagements. Requires extensive aerospace or defense experience, regional customer expertise, cross-cultural fluency, knowledge of U.S. and international defense sales frameworks, export controls, local partnerships, and English proficiency; Spanish and Portuguese are preferred. Top Skills: CRMDirect Commercial Sales (Dcs)EarForeign Military Sales (Fms)ItarPipeline Management ToolsUncrewed Air Systems

Remote Hiring Remotely in US 260K-320K Annually Senior level Remote Hiring Remotely in US 260K-320K Annually Senior level Leads development of agent evaluation systems that assess live software agents using trace data, rubrics, benchmarks, and LLM-as-a-judge techniques. Owns the evaluation platform’s full stack, creates reusable workflows and integrations, tracks emerging evaluation methods, and collaborates with data science teams. Manages and mentors engineers while promoting evaluation practices across agent-focused products. The summary above was generated by AI, An Hour Ago Remote or Hybrid 173K-312K Annually Senior level 173K-312K Annually Senior level Fintech * Payments * Software * Financial Services Lead Sales Development across North America and the UK, owning top-of-funnel pipeline, regional alignment, performance reporting, upmarket expansion, cross-functional partnerships, talent development, and technology-enabled productivity. The role manages teams and managers, establishes scalable inbound and outbound motions, partners with Marketing, Sales, Finance, and Strategy, and drives revenue-focused planning, qualification, and conversion. Top Skills: AIAnalyticsAutomationReporting Tools Dynatrace

Account Executive

3 Hours Ago Remote or Hybrid 140K-180K Annually Senior level 140K-180K Annually Senior level Artificial Intelligence * Big Data * Cloud * Information Technology * Software * Big Data Analytics * Automation Manage a named territory of ~18 accounts (7-10 customers, 5-8 prospects). Drive land-and-expand enterprise sales, acquire new logos, upsell/cross-sell Dynatrace observability solutions, consult with VP/C-level stakeholders, coordinate with sales engineering/partners, and ensure successful customer implementations. Top Skills: AWSDynatraceDynatrace IntelligenceGCPMeddpicAzureObservability Boeing

Business Development Specialist - Latin America - Remote

3 Hours Ago In-Office or Remote 159K-218K Annually Expert/Leader 159K-218K Annually Expert/Leader

G2 is the world’s largest and most trusted software marketplace. When you join G2, you’re joining the industry’s leading team that helps businesses reach their peak potential by powering decisions and strategies with trusted insights from real software users.

Now, we have joined forces with Capterra, SoftwareAdvice, and GetApp to create the largest source of online data and software insights to fuel intelligent buying in the age of AI. With 200M+ combined annual visitors and 6M verified reviews, we are now the centralized place to enable software buyers to make better and faster decisions with confidence.

And we are just getting started! We are setting out to transform the global B2B software industry and become the most trusted data foundation for buyers and sellers of software for the age of AI.

Does that sound exciting to you? Come join us as we try to reach our next PEAK!

About G2 - Our People

At G2, everything we are and what we do is grounded in our PEAK values- (Performance + Entrepreneurship + Authenticity + Kindness. Working at G2 means you are part of a value-driven, growing global community that climbs PEAKs together. We cheer for each other’s successes, learn from our mistakes, and support and lean on one another during challenging times. With ambition and entrepreneurial spirit we push each other to take on challenging work, which will help us all to grow and learn.

You will be part of a global, diverse team of smart, dedicated, and kind individuals - each with unique talents, aspirations, and life experiences. At the heart of our community and culture are our people-led ERGs, which celebrate and highlight the diverse identities of our global team. As an organization, we are intentional about our DEI and philanthropic work (like our G2 Gives program) because it encourages us all to be better people., With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.

Key Facts About Colorado Tech

• Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
• Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
• Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
• Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
• Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
• Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute