Agents Can't Iterate Against Tests That Lie

Coding agents write nearly all of your code. Now you need to prove it works.

Flaky end-to-end (E2E) tests in shared environments make that proof difficult and unreliable. With limited context, agents often reach for the wrong fix: increase the timeout or add a retry.

This talk is a practical case study in rebuilding trust in tests for AI-heavy engineering organizations. I’ll show the workflow that helped us reduce the share of PRs affected by E2E flakes from 100% to under 15% in six weeks, including the open-source libraries and agent skills we built to classify flaky tests, connect to observability signals, and decide which tests to keep, delete, or move down the pyramid.

Attendees will leave with a repeatable playbook for making coding agents safe to use at scale and for teaching them to root-cause failing tests rather than “fix” them with retries.