WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe • Jul 10, 2026 • Session details

Testing AI Agents: Automated Evaluation for Chatbots & RAG Systems

Sebastian Messingfeld

Traditional string-matching tests are obsolete for non-deterministic AI agents. Discover how to build an AI testing pyramid using LLM-as-a-judge to catch silent chatbot regressions in your deployment pipelines.

Pause
Mute Enter Fullscreen
#1 about 9 min

Understanding AI chatbot vulnerabilities and stateful attacks

How real-world AI agents silently fail through prompt injections, outdated knowledge, and unintended system manipulation.

#2 about 3 min

The AI agent test pyramid and basic validations

Structuring efficient test suites with cheap schema checks, reference-based assertions, and deterministic unit tests.

#3 about 4 min

Introducing LLMs as judges for automated testing

Using independent language models to critically review chatbot outputs at scale for correctness and consistency.

#4 about 7 min

Evaluating AI outputs practically using the DeepEval framework

Using the DeepEval Python framework to score system responses against retrieved context via specialized metrics.

#5 about 2 min

Running automated AI testing continuously in CI environments

Scheduling periodic test runs in integration pipelines to guard against model drift and knowledge base changes.

#6 about 4 min

Empowering non-developers with BDD and Gherkin testing

Enabling product owners and content editors to define quality expectations through human-readable structured test scenarios.

#7 about 4 min

Key takeaways for reliable AI agent testing

A summarized blueprint for preventing regressions and maintaining cross-functional ownership of overall chatbot quality.

Matching moments

2:11 min

Applying the testing pyramid to AI failures

Eric Deandrea Eric Deandrea · World Congress 2026 North America

1:56 min

Introduction to evaluating AI code review agents

Sofia Rest Sofia Rest · World Congress 2026 North America

2:28 min

Introduction to building reliable AI agents in production

Max Tkacz Max Tkacz · World Congress 2025

4:07 min

Running automated evaluation tests for conversational tool selection

Rishabh Budhiraja Rishabh Budhiraja +1 · World Congress 2026 Europe

2:06 min

Rethinking team structures around AI agent capabilities

Mike Mike · World Congress 2025

5:27 min

Evaluating AI agents through unpredictable behavior and logic tests

Chris Heilmann Chris Heilmann +2 · LIVE

More from World Congress 2026 Europe

Building the Nervous System of AI - Michael Kagan (NVIDIA) thumbnail
29:17
3 months ago

Building the Nervous System of AI - Michael Kagan (NVIDIA)

Michael Kagan, Sead Ahmetović

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote thumbnail
19:41
3 months ago

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote

Christian Heilmann

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend thumbnail
32:06
3 months ago

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

David vonThenen

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes thumbnail
35:13
3 months ago

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes

Irina Branovic

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬 thumbnail
24:23
3 months ago

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬

Jeremy Murray

Angstfreude - AI and Corporate Culture - The Thrill and the Threat thumbnail
30:26
3 months ago

Angstfreude - AI and Corporate Culture - The Thrill and the Threat

Axel Ebert, Karin Krobath

Upcoming sessions on this topic

Open session

Teaching AI Coding Agents to Build It Right the First Time

  • Sajeetharan Sinnathurai

    Microsoft

    Principal Product Manager

Open session

Build a Zero-Cost AI Agent in Your Browser

  • Aishwarya Mathuria

    Adobe

    Computer Scientist

  • Nishant Thakur

    Adobe

    Senior Computer Scientist

Open session

Architecting Trusted AI: Coupling Multi-Agent Systems with dbt Semantic Store for Zero-Hallucination

  • Pushkar Mishra

    J.P. Morgan

    Senior Vice President

Open session

Java is Coming for Python’s Lunch: The Rise of JVM AI Agents

  • Frederik Pietzko

    Jetbrains

    Developer Advocate

Open session

Beware of Strangers Bearing Code: Open Source Trust in the Agent Era

  • Vikram Vaswani

    Consultant

Open session

Prompt Debt: Why Your AI System Is Frozen in Time

  • Suriya Prakash B

    Red Hat

    Principal Software Engineer

Related videos

Who Tests the AI? Building Trustworthy AI Systems at Enterprise Scale thumbnail
27:57
World Congress 2026 North America • 18 days ago

Who Tests the AI? Building Trustworthy AI Systems at Enterprise Scale

Him Raj Singh

Reinventing Testing Practices in the AI Era thumbnail
30:19
World Congress 2026 North America • 18 days ago

Reinventing Testing Practices in the AI Era

Eric Deandrea

AI as a Test Designer: Transforming Experience into Automated Testing thumbnail
30:46
Europe 2026 Virtual • 3 months ago

AI as a Test Designer: Transforming Experience into Automated Testing

Alisa Hrustic

Beyond Chatbots: How to build Agentic AI systems thumbnail
24:22
World Congress 2025 • about 1 year ago

Beyond Chatbots: How to build Agentic AI systems

Philipp Schmid

Test Before Release, Enforce at Runtime: Governance for Tool-Using AI Agents thumbnail
21:16
World Congress 2026 North America • 18 days ago

Test Before Release, Enforce at Runtime: Governance for Tool-Using AI Agents

Sachin Gupta

AI Agents & Agentic AI thumbnail
26:53
Europe 2026 Virtual • 3 months ago

AI Agents & Agentic AI

Xavier Sala Presas

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.