> Markdown version of [/events/conference-india-2026/sessions/2012-the-pr-is-the-eval](https://www.wearedevelopers.com/events/conference-india-2026/sessions/2012-the-pr-is-the-eval). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# The PR Is the Eval


- **Event:** Conference India 2026

## Description

Most coding agents are evaluated at the moment they answer. Software teams discover whether an answer was useful much later: a test fails, a reviewer requests changes, the pull request stalls, or the code ships. This talk shows how to turn those delayed signals into a continuous evaluation system. We instrument AI conversations, tool calls, diffs, commits, CI runs, review comments, and merge outcomes, then join them into one trace from prompt to production.

Using a working implementation, I’ll walk through the event model, correlation logic, and queries that separate accepted work from rework and repeated failure. I’ll show the hard parts: carrying trace context across unrelated tools, distinguishing flaky CI from agent errors, protecting developer privacy, and avoiding vanity metrics. Finally, we’ll turn recurring corrections into human-approved candidate skills and test whether they improve first-pass CI, review rounds, and time-to-merge.

Attendees will leave with a product-agnostic blueprint for outcome-based evaluation of coding agents.

## Speaker

### [Jack Fitzsimons](https://www.wearedevelopers.com/@jack-fitzsimons)

CTO


## Related talks at this congress

- [Teaching AI Coding Agents to Build It Right the First Time](https://www.wearedevelopers.com/events/conference-india-2026/sessions/1995-teaching-ai-coding) — Sajeetharan Sinnathurai
- [Beware of Strangers Bearing Code: Open Source Trust in the Agent Era](https://www.wearedevelopers.com/events/conference-india-2026/sessions/1978-beware-of-strangers) — Vikram Vaswani
- [React Under Fire: Rendering Performance for Streaming Agent UIs](https://www.wearedevelopers.com/events/conference-india-2026/sessions/1992-react-under-fire) — Abhirupa Mitra
- [What I Got Wrong Shipping an MCP Server for Live Infrastructure](https://www.wearedevelopers.com/events/conference-india-2026/sessions/1999-what-i-got-wrong) — Pritesh Kiri

## Watch remotely

Can’t make it to Bengaluru? Watch this session live with Pro. You also get:

- All full videos, bookmarks, and playlists
- World Congress livestreams

[See pricing](https://www.wearedevelopers.com/pricing)

## Links

- [Get tickets](https://www.wearedevelopers.com/conference-india/tickets)