Session

Taming Rogue Agents: Observability-Driven Evaluation for Production Reliability

with Anjana Umapathy & Anagha Rumade & Apoorva Jaiswal

About This Session

Moving from a viral demo to reliable enterprise AI is the hardest hurdle in modern software engineering. Because agents are inherently non-deterministic, validating performance requires a granular look at the "why" and "how" behind every action. This session explores how observability redefines agentic evaluation beyond binary pass/fail testing, shifting the focus from the result to the entire reasoning chain. Featuring a live demo using Arize Phoenix and LangGraph, we’ll showcase how to trace complex loops and automate evaluations in real-time. What you’ll learn: Beyond Output: Why judging final answers alone is a recipe for silent failure. Live Implementation: Using Arize Phoenix with LangGraph to visualize traces and debug agentic "thought processes". The Playbook: An incremental framework for evaluation-first development. Are you ready to stop guessing and start measuring? Join us to master the art of agent evaluation.

Topics

Agents
Agentic AI
Generative AI (GenAI)
LangChain
Large Language Models (LLMs)
Multi-Agent Systems
Observability
OpenAI
Python

← Back to Schedule

Speakers

Anjana Umapathy

Applied AI/ML Lead and VP · JPMorgan Chase

Applied AI/ML Lead and VP at JPMorgan Chase

Read bio Hide bio

Anjana Umapathy is an Applied AI & ML Tech Lead and Manager in Global Banking at JPMorgan Chase, where she drives GenAI and Agentic AI intiatives. She holds an M.S. in Machine Learning & Natural Language Processing from Carnegie Mellon University and a B.Tech in Computer Science Engineering from PES University, India. Early in her career, Anjana honed her data science and software engineering skills at Adobe and SAP Concur. At JPMC, she has architected and developed the firm’s first Agentic AI solution - streamlining complex workflows and delivering measurable business impact. A prolific inventor and author, Anjana holds numerous publications and patents, both within and beyond JPMC. Anjana is passionate about mentorship and fostering diversity in tech and regularly advises in internal women’s networks. LinkedIn: https://www.linkedin.com/in/anjanaumapathy/ Google Scholar: https://scholar.google.com/citations?user=cKMbToEAAAAJ&hl=en
Anagha Rumade

Applied AI/ML Lead · JPMorgan Chase

Applied AI/ML Lead at JPMorgan Chase

Read bio Hide bio

Anagha Rumade is an Applied AI/ML Lead pioneering production-scale generative AI systems at the intersection of search, natural language processing, and large language models. She currently leads a machine learning team architecting an innovative internal search chatbot system powered by sophisticated LLM workflows, bringing cutting-edge retrieval and agentic AI to thousands of users. A recognized thought leader in AI/ML, Anagha was a featured speaker at Grace Hopper Celebration 2025 and holds multiple patents as an inventor in the AI/ML field. Her expertise spans GenAI, NLP, LLMs, MLOps, large-scale data engineering, and team leadership. Beyond her technical achievements, Anagha is deeply committed to expanding access to technology—dedicating her time to teaching underprivileged children and empowering the next generation of technologists. She brings both deep technical expertise and a passion for inclusive innovation to every stage and classroom. LinkedIn: https://www.linkedin.com/in/anagharumade/ Google Scholar: https://scholar.google.com/scholar?as_vis=0&q=Anagha+Rumade&hl=en&as_sdt=7,33
Apoorva Jaiswal

Applied AI/ML Lead · JPMorgan Chase

Applied AI/ML Lead at JPMorgan Chase

Read bio Hide bio

Apoorva Jaiswal is an Applied AI Lead at JPMorgan Chase focused on translating research into production systems by developing graph machine learning (graph ML) and LLM-powered agentic systems that serve Global Banking. She began her career at JPMC India in 2018 as a Software Engineer, building a strong foundation in systems and software design. In 2020, she pivoted to the India-based AI team, architecting cloud infrastructure to deploy ML models at scale before moving into hands-on model development. In 2023, she relocated to the US-based AI team and advanced into her current leadership role, leading cross-functional delivery of graph ML and agentic AI capabilities. Apoorva is a strong advocate of continuous learning. She earned her MS in Computer Science from the University of Massachusetts Amherst while working full-time. She contributes to the broader AI community through mentorship, teaching and talks: she has led workshops at the Grace Hopper Celebration on LangChain-based multimodal workflows (2024) and on LangGraph-based multi-agent workflows (2025), and has spoken at AI meetups in Silicon Valley. She organizes AI Meetups at the Silicon Valley Tech Center, JPMC and is a BobaTalks mentor. She holds patents in AI/ML and brings a global perspective from building and scaling production AI systems across India and the United States. Linkedin: https://www.linkedin.com/in/apoorvajaiswal Google Scholar: https://scholar.google.com/citations?hl=en&user=hLJgexgAAAAJ&view_op=list_works