The Illusion of Green: Why AI-Generated Browser Automation Tests Pass Locally but Fail in CI

Selenium doesn’t mock its own tests. It executes them against real browser and OS combinations. This exhaustive approach ensures the framework is reliable before code merges. When a test fails on CI(Continuous Integration), debugging it teaches you how the browser automation ecosystem actually behaves.

Over years contributing to Selenium, certain failure patterns emerged consistently: browser environment mismatches, cross-browser behaviour differences, spec interpretation gaps across browser vendors, inconsistencies between headless and headed modes, element synchronization issues (waiting for elements to be present, visible, or interactable), CI-specific constraints (CPU, memory, network variability), and test isolation failure.

These problems aren’t new. But AI-assisted testing makes them common again since the implicit context that comes from real debugging is missing. As a result, generative AI churns out tests that pass locally but fail in CI. When the CI error gets pasted back to AI, it regenerates code without understanding the root cause, often sending things in a different or frustrating direction. AI isn’t at fault, it’s simply skipping the learning journey developers take when debugging.

This talk breaks down each failure category and surfaces the implicit knowledge behind them, the kind that usually takes years to build. With this, the patterns become recognizable almost immediately and applicable to your own tests. More importantly, you will learn how to bring this context into AI-assisted testing so the same failures don’t keep resurfacing.