> Markdown version of [/jobs/ext/4023185-staff-software-engineer-testing-evals-frameworks](https://www.wearedevelopers.com/jobs/ext/4023185-staff-software-engineer-testing-evals-frameworks). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Staff Software Engineer - Testing & Evals Frameworks - **Company:** AI-Assisted High Performance Coaching, LLC - **Location:** McLean, VA, United States - **Experience:** Expert - **Salary:** $174,000.0 - $229,000.0 - **Contract:** Permanent contract - **Skills:** LangGraph Framework, Java (Programming Language), JavaScript (Programming Language), Application Programming Interfaces (APIs), Artificial Intelligence, Automation of Tests, Code Review, Continuous Integration, Software Debugging, Distributed Systems, Fraud Prevention and Detection, Fuzz Testing, Github, Identity and Access Management, Python (Programming Language), Key Management, Load Testing, Machine Learning, OAuth, OpenID, Ping (Networking Utility), Ruby on Rails, Role-Based Access Control, Reliability Engineering, Security Assertion Markup Language (SAML), System Testing, Strategies of Testing, TypeScript, Web Applications, Workflow Management Systems, Datadog, Google Cloud, Okta, LangChain, Retrieval-Augmented Generation, Large Language Models, Claude Code, Spring-boot, Gatling, Backend, Event Driven Architecture, Integration Tests, Kubernetes, Playwright, CrewAI, Front End Software Development, Virtual Agents, Restful APIs, Docker, Microservices - **Published:** October 30, 2026 - **Apply:** https://startup.jobs/staff-software-engineer-testing-evals-frameworks-idme-10611193 ## About the Role * 10+ years of experience (or equivalent: 10+ years with a Master's degree, or a PhD with 5+ years) as a software engineer in modern web applications and distributed systems. * Proficiency in Python/TypeScript/JavaScript/Java. * Hands-on experience with Playwright, Jest, API automation, or evaluation frameworks for software/AI systems. * Experience integrating automated tests into CI/CD pipelines (e.g., GitHub Actions, ArgoCD). * Proven ability to lead cross-team technical initiatives and influence engineering standards. * Track record of mentoring senior engineers and raising the quality bar across teams. * Strong written and verbal communication across engineering, product, and leadership., * API & Systems Testing: REST APIs, microservices, distributed and event-driven architectures; workflow orchestration (e.g., Temporal). * Identity & Access Management: OAuth 2.0, OIDC, SAML 2.0, SCIM 2.0; enterprise IdPs (Okta, Entra ID, Auth0, Ping); RBAC/ReBAC (Zanzibar, SpiceDB) and multi-tenant authorization. * Security & Compliance: AuthN/authZ testing, secrets management, tenant isolation, least-privilege access; IDV/KYC and PII/compliance validation. * Testing Strategy & Ops: Canary/phased rollouts, automated rollback, contract testing (Pact), performance/load testing (Locust, k6, Gatling). * Engineering & Platform: Java/Spring Boot or Ruby on Rails; GCP, Docker, Kubernetes; observability and chaos engineering. * AI-Augmented Testing: Using/building agentic tools for test generation, code review, and test planning. * AI Agent Evals: Experience building eval frameworks for LLM agents (e.g., correctness graders, LLM-as-judge, human-in-the-loop evals, benchmark dataset curation); familiarity with agentic frameworks (Claude API/Anthropic SDK, BrainTrust, LangChain, LangGraph, CrewAI, or similar); production monitoring for AI systems (behavioral drift detection, output sampling, shadow scoring); red-teaming or adversarial testing experience for AI models or agents; experience with RAG pipelines, including data chunking and pre-retrieval data quality gates. * Background in identity verification, fraud detection, or regulated industries. * Experience with observability tooling (Datadog, OpenTelemetry). ## Description This Staff Engineer role sits at the intersection of engineering, testing, AI evals, and developer experience. You will own the architecture and strategy for the evaluation frameworks, automation tools, and testing infrastructure that give ID.me engineering teams the confidence to ship quickly and safely across functional, end-to-end, LLM/eval, and security-focused testing practices, as well as the CI/CD quality gates the org depends on. It requires deep engineering rigor, the judgment to set testing strategy for complex, asynchronous, event-driven systems, and the ability to build tooling and standards that other teams adopt widely. You will raise the quality bar across the engineering org, mentor senior and mid-level engineers, champion AI-assisted testing practices, and partner directly with engineering, product, and security teams to embed quality into every stage of development for a platform that verifies the identity of millions of users., * Own Test Architecture & Strategy: Set the architecture and long-term strategy for automation frameworks and testing infrastructure (Playwright, Python, Jest, API automation) across ID.me's services, and ensure comprehensive, reliable coverage org-wide. * Lead Testing Strategy for Distributed Systems: Define end-to-end and integration testing strategy for complex, asynchronous, event-driven, and eventually-consistent systems, including workflow orchestration platforms such as Temporal. * Drive Testing-in-Production as an Org Standard: Build and champion canary deployments, progressive/phased rollouts, synthetic monitoring, and automated rollback triggers to validate real-world behavior safely before full release. * Own Security-Focused Test Automation: Set the standard for testing authentication/authorization flows (OAuth 2.0, OIDC), RBAC/ReBAC, secrets management, and tenant isolation. * CI/CD Quality Gates: Integrate automated testing into CI/CD pipelines (e.g., GitHub Actions) org-wide to give every team fast, reliable feedback. * Drive AI & Evals Culture: Build robust evaluation pipelines for AI-assisted features and LLM agents, while applying AI tools to accelerate test authoring, test plan generation, and code review across the org. * Cross-Team Collaboration: Partner with Security, Platform, and Product teams as a testing strategist, advising on testable code structure, coverage gaps, and quality checks throughout the development lifecycle. * Mentorship: Guide senior and mid-level engineers on test architecture, testing strategy, and quality practices. * Debug & Root Cause: Debug and root-cause issues across frontend, backend, APIs, and distributed services, and drive fixes to closure. * Reliability & Eval Metrics: Define and track reliability, quality, and eval accuracy metrics (e.g., time to detect, time to remediate, output quality scores) for the systems you evaluate and test.