We are looking for an AI Evaluation & Test Engineer to ensure generative AI models and applications are safe, accurate, trustworthy, and deliver an elegant user experience.

Validates AI models and agents for accuracy, safety, bias, and performance through structured testing, benchmarking, and continuous evaluation pipelines and will be responsible for following:

• Build and maintain AI evaluation pipelines to test, measure, and evaluate the behavior and performance of AI systems.
• Implement traces, spans, and session tracking for observability and identify error propagation in multi-step pipelines.
• Define AI quality metrics and KPIs around factuality, faithfulness, toxicity, grounding precision/recall, latency, cost, etc., with clear acceptance bars.
• Implement evaluation and testing automation to enable end-to-end system and regression testing at scale.
• Define criteria for and implement release gates in the CI/CD pipeline.
• Define criteria for and implement release gates in the CI/CD pipeline.
• Find creative ways to break products.
• Assist in root cause analysis and troubleshooting of bugs and field issues.
• Collaborate with cross-functional teammates from product, engineering, linguistics,, and customer support to shape human-AI interaction paradigms and ensure that our AI models and applications deliver the desired outcome and user experience.

Platform & Enablement Roles

• AI Platform Admin (M365, copilot Studio) Manages AI platforms and environments, including access provisioning, governance controls, and policy enforcement (e.g., DLP, security, and compliance).
• AI Reusable Utility Develops reusable components (e.g., prompts, connectors, APIs, templates) to accelerate AI solution delivery and promote standardization across use cases.
• AI Common Infrastructure, Framework & Observability Architect (AWS and Azure) Designs and maintains the foundational AI infrastructure, frameworks, and observability capabilities (telemetry, monitoring, metrics) required for scalable, reliable, and governed AI operations.

Core Responsibilities

• Adversarial Testing (Red Teaming): Design prompts to manipulate agent behavior, stress-test edge cases, and expose security vulnerabilities (e.g., prompt injection or PII leakage) before deployment.
• Pipeline Automation: Build and maintain automated regression testing, CI/CD release gates, and testing data sets (golden sets) to measure system drift.
• Grader Development: Implement “LLM-as-a-judge” frameworks, rule-based checks, and human-in-the-loop scoring rubrics to objectively evaluate open-ended AI outputs.
• Root Cause Analysis: Trace multi-turn conversations and agent tool interactions to diagnose when and why the AI chose the wrong path.
• Metric Definition: Establish and monitor AI KPIs such as factual accuracy, latency, cost, and grounding precision., Compliance with Client’s responsible AI principles and Acceptable Use policy
• Adherence to data residency, privacy (GDPR, HIPAA where applicable), and 21 CFR Part 11 controls where in scope
• Third-party risk assessment and SOC 2 Type II (or equivalent) certification
• Disclosure of subcontractors and offshore delivery locations
• Disclosure of model providers, training data practices, and any use of client data for model improvement (opt-out required)

LI-NorthAmerica

About NTT DATA

NTT DATA is a $30 billion business and technology services leader, serving 75% of the Fortune Global 100. We are committed to accelerating client success and positively impacting society through responsible innovation. We are one of the world’’s leading AI and digital infrastructure providers, with unmatched capabilities in enterprise-scale AI, cloud, security, connectivity, data centers and application services. our consulting and Industry solutions help organizations and society move confidently and sustainably into the digital future. As a Global Top Employer, we have experts in more than 50 countries. We also offer clients access to a robust ecosystem of innovation centers as well as established and start-up partners. NTT DATA is a part of NTT Group, which invests over $3 billion each year in R&D.

Whenever possible, we hire locally to NTT DATA offices or client sites. This ensures we can provide timely and effective support tailored to each client’s needs. While many positions offer remote or hybrid work options, these arrangements are subject to change based on client requirements. For employees near an NTT DATA office or client site, in-office attendance may be required for meetings or events, depending on business needs. At NTT DATA, we are committed to staying flexible and meeting the evolving needs of both our clients and employees. NTT DATA recruiters will never ask for payment or banking information and will only use @nttdata.com and @talent.nttdataservices.com email addresses. If you are requested to provide payment or disclose banking information, please submit a contact us form, https://us.nttdata.com/en/contact-us.

Artificial Intelligence (AI), Testing, Artificial Intelligence (AI), Testing, User Interface/Experience (UI/UX), Benchmarking, Artificial Intelligence (AI) Agents, Performance Testing, Model Validation, Quality Metrics, * Programming: 5+ years of strong proficiency in Python and testing frameworks like pytest.

• AI & LLM Frameworks: 5+ years of hands-on experience with evaluation tools like LangSmith, DeepEval, TruLens, or Promptfoo.
• Orchestration Tools: 3 to 5 years of familiarity with agentic workflows built on LangChain, CrewAI, or LlamaIndex.
• Observability: Understanding of tracing and session tracking to map how errors propagate in RAG systems.
• 5+ years of strong software testing fundamentals and expertise in writing test plans, executing test cases, and generating detailed reports and dashboards.
• Strong analytical and debugging skills, and attention to detail.
• 5+ years of proficiency in Python, scripting, and software testing automation frameworks and tools such as Pytest, Selenium, Robot Framework, etc.
• Working knowledge of generative AI models, AI agents, and related concepts such as retrieval augmented generation (RAG), prompt engineering, context engineering, explainability, traceability, observability, guard rails, reasoning, specificity, etc.
• Sound understanding of the fundamental differences in the approach for testing conventional software versus evaluating generative AI systems.
• Team player with excellent interpersonal skills and the ability to collaborate effectively with remote and cross- functional team members.
• Go-getter attitude and ability to flourish in a fast-paced, startup environment.
• Experience in any of the following would be a big plus.
• AI evaluation frameworks such as Arize, Braintrust, DeepEval, LangSmith, Ragas
• AI safety and red teaming experience, e.g., prompt injection, jailbreak, adversarial and stress testing.
• Different types of AI evaluation methods, e.g, Human-in-the-loop, LLM-as-a-Judge.

Typical Qualifications

• Experience: Usually 2-4+ years of hands-on experience as an ML Engineer, AI Engineer, or a specialized QA/Testing Engineer focusing on machine learning.
• Education: Degree in Computer Science, Data Science, Linguistics, or closely related technical fields