WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe • Jul 10, 2026 • Session details

Testing AI Agents: Automated Evaluation for Chatbots & RAG Systems

Sebastian Messingfeld

Traditional string-matching tests are obsolete for non-deterministic AI agents. Discover how to build an AI testing pyramid using LLM-as-a-judge to catch silent chatbot regressions in your deployment pipelines.

Pause
Mute Enter Fullscreen
#1 about 9 min

Understanding AI chatbot vulnerabilities and stateful attacks

How real-world AI agents silently fail through prompt injections, outdated knowledge, and unintended system manipulation.

#2 about 3 min

The AI agent test pyramid and basic validations

Structuring efficient test suites with cheap schema checks, reference-based assertions, and deterministic unit tests.

#3 about 4 min

Introducing LLMs as judges for automated testing

Using independent language models to critically review chatbot outputs at scale for correctness and consistency.

#4 about 7 min

Evaluating AI outputs practically using the DeepEval framework

Using the DeepEval Python framework to score system responses against retrieved context via specialized metrics.

#5 about 2 min

Running automated AI testing continuously in CI environments

Scheduling periodic test runs in integration pipelines to guard against model drift and knowledge base changes.

#6 about 4 min

Empowering non-developers with BDD and Gherkin testing

Enabling product owners and content editors to define quality expectations through human-readable structured test scenarios.

#7 about 4 min

Key takeaways for reliable AI agent testing

A summarized blueprint for preventing regressions and maintaining cross-functional ownership of overall chatbot quality.

Matching moments

2:28 min

Introduction to building reliable AI agents in production

Max Tkacz Max Tkacz · World Congress 2025

4:07 min

Running automated evaluation tests for conversational tool selection

Rishabh Budhiraja Rishabh Budhiraja +1 · World Congress 2026 Europe

2:06 min

Rethinking team structures around AI agent capabilities

Mike Mike · World Congress 2025

5:27 min

Evaluating AI agents through unpredictable behavior and logic tests

Chris Heilmann Chris Heilmann +2 · LIVE

2:40 min

Reviewing live performance of self-correcting AI engineering agents

Ingo Eichhorst Ingo Eichhorst · World Congress 2026 Europe

3:19 min

Validating code generation with evals, datasets, and graders

Przemysław Ładyński Przemysław Ładyński · World Congress 2026 Europe

More from World Congress 2026 Europe

Building the Nervous System of AI - Michael Kagan (NVIDIA) thumbnail
29:17
3 months ago

Building the Nervous System of AI - Michael Kagan (NVIDIA)

Michael Kagan, Sead Ahmetović

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote thumbnail
19:41
3 months ago

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote

Christian Heilmann

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend thumbnail
32:06
3 months ago

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

David vonThenen

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes thumbnail
35:13
3 months ago

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes

Irina Branovic

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬 thumbnail
24:23
3 months ago

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬

Jeremy Murray

Angstfreude - AI and Corporate Culture - The Thrill and the Threat thumbnail
30:26
3 months ago

Angstfreude - AI and Corporate Culture - The Thrill and the Threat

Axel Ebert, Karin Krobath

Related videos

AI as a Test Designer: Transforming Experience into Automated Testing thumbnail
30:46
Europe 2026 Virtual • 3 months ago

AI as a Test Designer: Transforming Experience into Automated Testing

Alisa Hrustic

Beyond Chatbots: How to build Agentic AI systems thumbnail
24:22
World Congress 2025 • about 1 year ago

Beyond Chatbots: How to build Agentic AI systems

Philipp Schmid

AI Agents & Agentic AI thumbnail
26:53
Europe 2026 Virtual • 3 months ago

AI Agents & Agentic AI

Xavier Sala Presas

Beyond the Benchmark: How to Evaluate AI Agents in the Real World thumbnail
30:04
World Congress 2026 Europe • 3 months ago

Beyond the Benchmark: How to Evaluate AI Agents in the Real World

Taylor Jordan Smith

Make it simple, using generative AI to accelerate learning thumbnail
26:54
World Congress 2024 • about 2 years ago

Make it simple, using generative AI to accelerate learning

Duan Lightfoot

Back to the Roots: Testing in the Age of AI thumbnail
29:50
World Congress 2026 Europe • 3 months ago

Back to the Roots: Testing in the Age of AI

Jakub Janczyk

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.