Taming Rogue Agents: Observability-Driven Evaluation for Production Reliability
Anagha Rumade
Applied AI/ML Lead · JPMorgan Chase
Anjana Umapathy
Applied AI/ML Lead & VP · JPMorgan Chase
Apoorva Jaiswal
Applied AI/ML Lead · JPMorgan Chase
What this session covers
Moving from a viral demo to reliable enterprise AI is the hardest hurdle in modern software engineering. Because agents are inherently non-deterministic, validating performance requires a granular look at the “why” and “how” behind every action.
This session explores how observability redefines agentic evaluation beyond binary pass/fail testing, shifting the focus from the result to the entire reasoning chain. Featuring a live demo using Arize Phoenix and LangGraph, we’ll showcase how to trace complex loops and automate evaluations in real-time.
What you’ll learn:
Beyond Output: Why judging final answers alone is a recipe for silent failure.
Live Implementation: Using Arize Phoenix with LangGraph to visualize traces and debug agentic “thought processes”.
The Playbook: An incremental framework for evaluation-first development.
Are you ready to stop guessing and start measuring? Join us to master the art of agent evaluation.
World Congress 2026 North America
Related talks
- Closing the Visibility Gap: Lessons from Safety Critical Agentic Systems
Vivek Pandit · Turing
Fri 25 Sep · Outdoor Stage
- Your Evals Passed. Your Agent Just Emptied a Database.
Tejas Pravinbhai Patel · Amazon
Fri 25 Sep · Stage 3
- Your Agents Need Observability Before They Need Better Models
Julia Furst Morgado · Dash0
Thu 24 Sep · Stage 5
- Evals Are Infra: Building AI Systems Developers Can Actually Trust
Phoebe Wang · OpenAI
Fri 25 Sep · Stage 4