> Markdown version of [/events/world-congress-2026-north-america/sessions/1793-21-experiments-in](https://www.wearedevelopers.com/events/world-congress-2026-north-america/sessions/1793-21-experiments-in). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# 21 Experiments in Six Weeks: A Playbook for Improving Your AI Agent


- **Event:** World Congress 2026 North America

## Description

Writing the code was the easy part. But setting out to improve Sentry’s AI code review agent meant answering a much harder question: what does it mean to make your agent “better”, and how do you know if it’s working? Over the course of six weeks, our team ran 21 live experiments to find out. Along the way, we built an eval loop that moved from offline datasets and limited observability to A/B testing infrastructure, composite metrics, and detailed cost tracking in a single dashboard. And our most honest signal—whether the developer actually fixed the flagged bug—surfaced results that broke our intuition (like why a smarter model isn’t always the answer) and forced hard tradeoffs between code review quality and cost. In this talk, I’ll walk through the practical playbook for improving your AI agent: how to define “better”, how to measure it, and how to get quality gains without breaking the bank.

## Speaker

### [Sofia Rest](https://www.wearedevelopers.com/@sofia-rest)

Software Engineer at Sentry


## Related talks at this congress

- [Agents Can't Iterate Against Tests That Lie](https://www.wearedevelopers.com/events/world-congress-2026-north-america/sessions/1700-agents-can-t-iterate) — Rocky Warren
- [Taming Rogue Agents: Observability-Driven Evaluation for Production Reliability](https://www.wearedevelopers.com/events/world-congress-2026-north-america/sessions/1768-taming-rogue-agents) — Anagha Rumade, Anjana Umapathy, Apoorva Jaiswal
- [AI That Argues With Itself: Building Self-Debating Systems That Catch Their Own Bugs](https://www.wearedevelopers.com/events/world-congress-2026-north-america/sessions/1401-ai-that-argues-with) — Shreya Singhal
- [Closing the Visibility Gap: Lessons from Safety Critical Agentic Systems](https://www.wearedevelopers.com/events/world-congress-2026-north-america/sessions/1408-closing-the) — Vivek Pandit

## Watch remotely

Can’t make it to San José? Watch this session live with Pro. You also get:

- All full videos, bookmarks, and playlists
- World Congress livestreams

[See pricing](https://www.wearedevelopers.com/pricing)

## Links

- [Get tickets](https://www.wearedevelopers.com/world-congress-north-america/tickets)