WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2025 • Aug 20, 2025 • Session details

Evaluating AI models for code comprehension

Merrill Lutsky

Are noisy AI code reviews eroding your engineers' trust? Discover the evaluation strategies that prove why Claude 3.7 Sonnet outperforms Gemini and GPT-4o for automated pull requests.

Pause
Mute Enter Fullscreen
#1 about 5 min

Scaling code review in the era of generated code

The rapid increase in code generation necessitates an improved outer loop for comprehensive testing and deployment.

#2 about 2 min

Automating pull request feedback with AI code review agents

Asynchronous bots integrated directly into version control can summarize and prioritize changes without causing developer fatigue.

#3 about 3 min

Measuring AI review quality through comment acceptance rates

The ultimate evaluation of automated feedback relies on tracking the percentage of suggestions that drive actual source code modifications.

#4 about 2 min

Why evaluations are the primary lever for language models

Unlike traditional machine learning pipelines, optimizing foundation models relies almost entirely on refining continuous evaluation datasets.

#5 about 2 min

Defining matched and unmatched comment scores for evaluations

Model accuracy is benchmarked by finding expected issues while strictly preventing noisy or hallucinated developer suggestions.

#6 about 1 min

Assessing GPT-4o performance for pull request feedback

GPT-4o provides precise and clean suggestions but struggles to identify all the expected necessary code changes.

#7 about 2 min

Performance trade-offs of the o3 reasoning model

Deeper reasoning models offer excellent recall but suffer from significantly higher latency during the asynchronous review process.

#8 about 2 min

Evaluating Gemini on context windows and suggestion noise

A massive context window allows for high issue discovery but often generates excessive and extraneous developer noise.

#9 about 3 min

Choosing Claude Sonnet 3.7 to balance signal and noise

Claude Sonnet 3.7 provides the best blend of expected issue detection without frustrating developers with irrelevant comments.

#10 about 3 min

Summarizing model benchmarks and the need for continuous evaluation

Maintaining competitive code comprehension requires frequent dataset updates as providers regularly release distinct architectural variations.

#11 about 2 min

Identifying performance gaps in newly released reasoning models

Early testing on novel model capabilities suggests that excessive reasoning cycles can occasionally degrade practical review performance.

Matching moments

2:40 min

Reviewing live performance of self-correcting AI engineering agents

Ingo Eichhorst Ingo Eichhorst · WWC Europe 2026

3:01 min

Balancing artificial intelligence tools with foundational software engineering skills

Tim Ruscica · Coffee With Developers

3:22 min

Evaluating advanced artificial intelligence platforms for daily recruitment

Rudi Bauer Rudi Bauer +1 · Cappuccino with HR

1:56 min

Analyzing cloud-based AI code completion architectures

Daniel Savenkov Daniel Savenkov · WWC 2024

1:49 min

Using a council of adversarial agents for code review

Milin Desai Milin Desai +1 · WWC Europe 2026

2:33 min

Evaluating AI comprehension and output quality

Perf + AI

More from World Congress 2025

GenAI Security: Navigating the Unseen Iceberg thumbnail
24:33
10 months ago

GenAI Security: Navigating the Unseen Iceberg

Maish Saidel-Keesing

Behind the Code: How Women Are Powering the Future of AI thumbnail
30:28
12 months ago

Behind the Code: How Women Are Powering the Future of AI

Alexandra Wudel, Laura Moritz, Madalina Florean

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise? thumbnail
28:12
12 months ago

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise?

Justus Lumpe, Konstantin Vinogradov, Nils Eiteneyer, Stephan Wirris

AI in Leadership: How Technology is Reshaping Executive Roles thumbnail
28:19
12 months ago

AI in Leadership: How Technology is Reshaping Executive Roles

Bernd Datler, Jasmin Kaiser, Jeff Hausmann, Sonja Alvarez

The AI-Ready Stack: Rethinking the Engineering Org of the Future thumbnail
28:09
12 months ago

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Alex Laubscher, Jan Oberhauser, Keno Dreßel, Mirko Novakovic

AI Sovereignty: What Does It Take? thumbnail
29:13
12 months ago

AI Sovereignty: What Does It Take?

Boris Hecker, Clemens Wasner, Robin Hermann, Sonja Alvarez

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Beyond Vibe Coding: Using CLI Tools as Your AI Counterpart in Pair Programming

Ron Veen

Java enthusiast and Special agent for Team Rockstars IT

Ron Veen
Open session

World Congress 2026 North America

No Single Model to Rule Them All: Building Resilient AI Agents Across Open & Closed LLMs

Emmanuel Acheampong

Senior Manager Developer Relations at Crusoe AI

Emmanuel Acheampong
Open session

World Congress 2026 North America

Evals Are Infra: Building AI Systems Developers Can Actually Trust

Phoebe Wang

Member of Technical Staff at OpenAI

Phoebe Wang
Open session

World Congress 2026 North America

There's no dark factory without better software verifiers

Dexter Horthy

Co-Founder, HumanLayer

Dexter Horthy
Open session

World Congress 2026 North America

AI That Argues With Itself: Building Self-Debating Systems That Catch Their Own Bugs

Shreya Singhal

AI Applied Scientist at Claritev

Shreya Singhal
Open session

World Congress 2026 North America

Ship 10x Faster: AI-Powered Development with Claude Code and MCP Tools

Viktoria Semaan

Principal Technical Evangelist at Databricks

Viktoria Semaan

Related videos

The Limits of Prompting: ArchitectingTrustworthy Coding Agents thumbnail
24:05
WWC 2025 • 12 months ago

The Limits of Prompting: ArchitectingTrustworthy Coding Agents

Nimrod Kor

How we built an AI-powered code reviewer in 80 hours thumbnail
28:07
WWC 2025 • 12 months ago

How we built an AI-powered code reviewer in 80 hours

Yan Cui

Teaching an LLM to review code … like a Senior Engineer! thumbnail
32:26
WWC Europe 2026 • about 1 month ago

Teaching an LLM to review code … like a Senior Engineer!

Kesha Mykhailov

Developer Productivity Using AI Tools and Services - Ryan J Salva thumbnail
59:16
Coffee With Developers • about 1 year ago

Developer Productivity Using AI Tools and Services - Ryan J Salva

Ryan J Salva

Beyond the IDE: A new era of agent collaboration thumbnail
27:12
WWC 2025 • 12 months ago

Beyond the IDE: A new era of agent collaboration

Ryan J. Salva

Innovating Developer Tools with AI: Insights from GitHub Next thumbnail
42:38
LIVE • over 1 year ago

Innovating Developer Tools with AI: Insights from GitHub Next

Krzystof Czieslak

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.