Testing AI Agents: Automated Evaluation for Chatbots & RAG Systems
-
Sebastian Messingfeld
Eurowings Digital
Staff Engineer
For two decades, QA engineers have written brittle selectors that shatter the moment a frontend developer renames a button class. We’ve accepted flaky tests, selector maintenance, and the endless cat-and-mouse game with dynamic IDs as the unavoidable cost of automation. AI browser agents like Skyvern and Browser-Use are challenging that assumption, navigating websites using vision and reasoning instead of XPaths, adapting when layouts change, and then compiling successful runs into Playwright scripts that execute without keeping an LLM in the loop. In this session, I’ll demonstrate live how Skyvern automates a real workflow that would traditionally require careful selector engineering. You’ll see the agent reason through the interface, handle unexpected states, and generate executable code. We’ll examine that generated code together, what’s surprisingly clean, what’s brittle in new ways, and what happens when the AI encounters edge cases that would trip up a human automator too. But this isn’t an obituary for Selenium or Playwright. It’s a thesis about architectural evolution. Classical frameworks aren’t dying; they’re becoming the invisible execution layer beneath AI agents the same way assembly didn’t disappear when we got high-level languages. The question isn’t whether to abandon your existing test suites, but how to position them in a world where AI handles the fragile parts. You’ll leave with a clear mental model for when AI-native automation wins, when traditional scripts remain superior, and how to position yourself as QA engineering transforms from selector maintenance to agent orchestration.
World Congress 2026 Europe
Sebastian Messingfeld
Eurowings Digital
Staff Engineer
Jakub Janczyk
Help Scout
Senior Engineer/Tech Lead
Dominik Srednicki
SAP
Principal Architect
Elena Lucarelli
SAP Signavio
Principal AI DevX Lead
Michael Friedrich
GitLab
Principal Developer Advocate