> Markdown version of [/events/conference-india-2026/sessions/2012-did-the-model](https://www.wearedevelopers.com/events/conference-india-2026/sessions/2012-did-the-model). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Did the Model Improve, or Just the Score? A Developer’s Guide to LLM Evaluation


- **Event:** Conference India 2026

## Description

A model can extract the right information and still score zero because its answer differs from a benchmark’s reference. Fine-tuning can raise that score, but does the improvement carry over to the inputs your application actually sees? This talk shows developers how to use imperfect benchmarks to make better decisions about model selection.

Using a working document-extraction example, I’ll walk through inspecting benchmark failures, separating model errors from scoring problems, and using semantic assessment to investigate disagreements. We’ll cover the limitations of LLM judges, the importance of preserving application requirements, and what individual examples can reveal that aggregate scores miss.

We’ll also explore how fine-tuning to benchmark data can produce strong scores, and what developers should check to assess whether that performance carries over to data outside the benchmark.

Attendees will leave with practical ways to assess benchmark claims, investigate whether model performance generalises, and make existing evaluation datasets more useful for choosing and improving models.

## Speaker

### [Jack Fitzsimons](https://www.wearedevelopers.com/@jack-fitzsimons)

CTO


## Related talks at this congress

- [Teaching AI Coding Agents to Build It Right the First Time](https://www.wearedevelopers.com/events/conference-india-2026/sessions/1995-teaching-ai-coding) — Sajeetharan Sinnathurai
- [Beware of Strangers Bearing Code: Open Source Trust in the Agent Era](https://www.wearedevelopers.com/events/conference-india-2026/sessions/1978-beware-of-strangers) — Vikram Vaswani
- [React Under Fire: Rendering Performance for Streaming Agent UIs](https://www.wearedevelopers.com/events/conference-india-2026/sessions/1992-react-under-fire) — Abhirupa Mitra
- [What I Got Wrong Shipping an MCP Server for Live Infrastructure](https://www.wearedevelopers.com/events/conference-india-2026/sessions/1999-what-i-got-wrong) — Pritesh Kiri

## Watch remotely

Can’t make it to Bengaluru? Watch this session live with Pro. You also get:

- All full videos, bookmarks, and playlists
- World Congress livestreams

[See pricing](https://www.wearedevelopers.com/pricing)

## Links

- [Get tickets](https://www.wearedevelopers.com/conference-india/tickets)