Agents Can't Iterate Against Tests That Lie

Between February 2025 and February 2026, coding agents went from writing none of Clipboard’s code to nearly all of it. That changed our bottleneck. Writing code got cheap, but proving it worked did not.

The failure mode was not just model quality. It was flaky E2E tests in shared environments that were hard to reproduce. With limited context, agents often reached for the wrong fix: increase the timeout or add a retry.

This talk is a practical case study in rebuilding trust in tests for AI-heavy engineering organizations. I will show the workflow that helped us reduce the share of PRs affected by E2E flakes from 100% to under 15% in six weeks, including the open source libraries and agent skills we built to classify flaky tests, connect to observability signals, and decide which tests to harden, delete, quarantine, or move down the pyramid.

Attendees will leave with a repeatable playbook for making coding agents safe to use at scale without teaching them that every failing test should be “fixed” with retries.