Senior AI Evaluation Engineer

· Design and operate end-to-end evaluation frameworks for agentic AI systems (offline regression, online monitoring, A/B testing)

· Define quality rubrics, scoring architectures, and evaluation standards across multi-agent workflows

· Build LLM-as-judge and Agent-as-judge pipelines assessing trajectory, tool-call accuracy, groundedness, and safety

· Own and deliver evaluation infrastructure end-to-end; mentor teams and uplevel customers on evals best practices

· GenAI Experience (Level 400) with Advanced Python

· Experience building production grade systems

· Hands-on experience with agent evaluation: offline/on-demand regression testing and online/continuous live-traffic scoring

· LLM-as-judge proficiency: rubric design, scoring at session, response, and tool-call granularity

· Core eval metrics: task completion, tool-call accuracy, trajectory quality, groundedness, safety

· Agentic framework experience (Strands, LangGraph, CrewAI or similar)

· AWS Bedrock (Models, Guardrails) + CloudWatch; Lambda, S3, IAM fundamentals

Should Have

· Bedrock AgentCore Evaluations (on-demand SDK runner, online monitoring, A/B testing)

· Agent-as-a-Judge framework; HITL evaluation pipelines

· Benchmarking frameworks (tau-bench, SWE-bench, GAIA) and custom dataset construction

· RAG quality evaluation (faithfulness, context precision, recall)

· CI/CD integration for eval regression; observability tooling (tracing, structured logging)

· Legacy technology familiarity (Java/Spring, Sybase, DB2, or COBOL) as source context for agentic modernization or integration pattern design

· Financial services or regulated-industry experience (compliance-aware delivery: trade surveillance, data controls, audit requirements)

· Large-scale portfolio delivery experience - reusable patterns or components applied across 50+ applications, including wave planning and throughput measurement

LTM is a global technology consulting and digital solutions company that enables enterprises across industries to reimagine business models, accelerate innovation, and maximize growth by harnessing digital technologies. As a digital transformation partner to more than 700+ clients, LTM brings extensive domain and technology expertise to help drive superior competitive differentiation, customer experiences, and business outcomes in a converging world. Powered by nearly 90,000 talented and entrepreneurial professionals across more than 30 countries, LTM a Larsen & Toubro Group company combines the industry-acclaimed strengths of erstwhile Larsen and Toubro Infotech and Mindtree in solving the most complex business challenges and delivering transformation at scale. For more information, please visit