AI Evaluation Infrastructure & Production Readiness

AI quality cannot be proven once at launch it is an ongoing discipline. Because AI systems are probabilistic, quality can shift as models, prompts, content, vendors, retrieval, and user behavior change. This role builds the evaluation discipline and tooling that produces repeatable evidence for whether an AI system is accurate, safe, compliant, grounded, and ready to scale. It is the enterprise’s most important AI risk-control role.

What You’ll Build :

• Evaluation harnesses and tooling that can be run repeatedly across AI systems.
• Golden test sets and scenario libraries covering expected behaviors and edge cases.
• Regression testing to catch quality changes when models, prompts, or content change.
• Hallucination testing and grounding/faithfulness testing.
• Bias and fairness testing to detect discriminatory or disparate-impact outcomes across protected classes, in support of fair-lending obligations (e.g., ECOA / Reg B).
• Adversarial and red-team testing, including jailbreak, prompt-injection, and harmful-output resistance.
• Policy-adherence testing against enterprise, compliance, and regulatory requirements.
• Drift detection and ongoing production (online) monitoring sampling live traffic, canary evaluations, and catching regressions after release.
• Human and subject-matter-expert evaluation workflows, plus curation, labeling, and versioning of evaluation datasets (including governance of any customer data they contain).
• Production-readiness gates that AI systems must pass before scaling.
• Evaluation evidence and reporting that AI governance and model-risk committees rely on to make go/no-go decisions.
• AI vendor acceptance criteria for third-party agents and solutions.

Why This Role Matters : Without a rigorous evaluation function, AI scales on the strength of demos, pilots, anecdotes, or vendor claims rather than repeatable evidence. This role creates the evidence system leadership needs to decide, with confidence, whether an AI system is ready. Evaluations apply equally to agents built in-house and to third-party agents delivered by vendors.

• 7+ years of relevant experience in software quality, data science, machine learning, or a closely related field, including experience leading a technical workstream.
• Demonstrated experience designing evaluation methods for LLM or ML systems (accuracy, grounding, safety, and policy adherence).
• Strong grasp of testing methodology: test-set design, regression testing, and metrics that hold up over time.
• Statistical rigor: experimental design, significance testing, and sample sizing.
• Experience with bias and fairness evaluation; familiarity with fair-lending concepts (ECOA / Reg B,disparate impact) is a strong plus.
• Ability to define and enforce production-readiness gates and acceptance criteria.
• Comfort partnering with risk, compliance, and legal to translate requirements into testable criteria., * Experience building evaluation harnesses, LLM-as-judge pipelines, or automated eval frameworks.
• Familiarity with hallucination, faithfulness, and drift-detection techniques.
• Experience setting vendor acceptance criteria and evaluating third-party AI solutions.
• Model risk management or model validation background (e.g., SR 11-7-aligned practices).
• Background in regulated financial services.