WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2025 • Aug 20, 2025 • Session details

Evaluating AI models for code comprehension

Merrill Lutsky

Are noisy AI code reviews eroding your engineers' trust? Discover the evaluation strategies that prove why Claude 3.7 Sonnet outperforms Gemini and GPT-4o for automated pull requests.

Pause
Mute Enter Fullscreen
#1 about 5 min

Scaling code review in the era of generated code

The rapid increase in code generation necessitates an improved outer loop for comprehensive testing and deployment.

#2 about 2 min

Automating pull request feedback with AI code review agents

Asynchronous bots integrated directly into version control can summarize and prioritize changes without causing developer fatigue.

#3 about 3 min

Measuring AI review quality through comment acceptance rates

The ultimate evaluation of automated feedback relies on tracking the percentage of suggestions that drive actual source code modifications.

#4 about 2 min

Why evaluations are the primary lever for language models

Unlike traditional machine learning pipelines, optimizing foundation models relies almost entirely on refining continuous evaluation datasets.

#5 about 2 min

Defining matched and unmatched comment scores for evaluations

Model accuracy is benchmarked by finding expected issues while strictly preventing noisy or hallucinated developer suggestions.

#6 about 1 min

Assessing GPT-4o performance for pull request feedback

GPT-4o provides precise and clean suggestions but struggles to identify all the expected necessary code changes.

#7 about 2 min

Performance trade-offs of the o3 reasoning model

Deeper reasoning models offer excellent recall but suffer from significantly higher latency during the asynchronous review process.

#8 about 2 min

Evaluating Gemini on context windows and suggestion noise

A massive context window allows for high issue discovery but often generates excessive and extraneous developer noise.

#9 about 3 min

Choosing Claude Sonnet 3.7 to balance signal and noise

Claude Sonnet 3.7 provides the best blend of expected issue detection without frustrating developers with irrelevant comments.

#10 about 3 min

Summarizing model benchmarks and the need for continuous evaluation

Maintaining competitive code comprehension requires frequent dataset updates as providers regularly release distinct architectural variations.

#11 about 2 min

Identifying performance gaps in newly released reasoning models

Early testing on novel model capabilities suggests that excessive reasoning cycles can occasionally degrade practical review performance.

Matching moments

2:40 min

Reviewing live performance of self-correcting AI engineering agents

Ingo Eichhorst Ingo Eichhorst · World Congress 2026 Europe

3:01 min

Balancing artificial intelligence tools with foundational software engineering skills

Tim Ruscica · Coffee With Developers

3:22 min

Evaluating advanced artificial intelligence platforms for daily recruitment

Rudi Bauer Rudi Bauer +1 · Cappuccino with HR

1:56 min

Analyzing cloud-based AI code completion architectures

Daniel Savenkov Daniel Savenkov · World Congress 2024

1:49 min

Using a council of adversarial agents for code review

Milin Desai Milin Desai +1 · World Congress 2026 Europe

2:33 min

Evaluating AI comprehension and output quality

Perf + AI

More from World Congress 2025

GenAI Security: Navigating the Unseen Iceberg thumbnail
24:33
11 months ago

GenAI Security: Navigating the Unseen Iceberg

Maish Saidel-Keesing

Behind the Code: How Women Are Powering the Future of AI thumbnail
30:28
about 1 year ago

Behind the Code: How Women Are Powering the Future of AI

Alexandra Wudel, Laura Moritz, Madalina Florean

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise? thumbnail
28:12
about 1 year ago

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise?

Justus Lumpe, Konstantin Vinogradov, Nils Eiteneyer, Stephan Wirris

AI in Leadership: How Technology is Reshaping Executive Roles thumbnail
28:19
about 1 year ago

AI in Leadership: How Technology is Reshaping Executive Roles

Bernd Datler, Jasmin Kaiser, Jeff Hausmann, Sonja Alvarez

The AI-Ready Stack: Rethinking the Engineering Org of the Future thumbnail
28:09
about 1 year ago

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Alex Laubscher, Jan Oberhauser, Keno Dreßel, Mirko Novakovic

AI Sovereignty: What Does It Take? thumbnail
29:13
about 1 year ago

AI Sovereignty: What Does It Take?

Boris Hecker, Clemens Wasner, Robin Hermann, Sonja Alvarez

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 24, 2026 · 15:30–16:00

Stage 5

The reviewer can't be the author: independent verification for AI-generated code

Manish Kapur

VP, Product and Solutions at Sonar

Manish Kapur
Open session

World Congress 2026 North America

September 24, 2026 · 15:30–16:00

Stage 7

Beyond Vibe Coding: Using CLI Tools as Your AI Counterpart in Pair Programming

Ron Veen

Java enthusiast and Special agent for Team Rockstars IT

Ron Veen
Open session

World Congress 2026 North America

September 24, 2026 · 16:10–16:40

Stage 2

The Death of the Code Review

Laurie Voss

Head of Developer Relations

Laurie Voss
Open session

World Congress 2026 North America

September 25, 2026 · 14:50–15:20

Tech Leaders Stage

Keeping Code Quality at AI Speed

Daksh Gupta, Arthur Hicken, Brian Mann

Daksh Gupta
Arthur Hicken
Brian Mann
Open session

World Congress 2026 North America

September 23, 2026 · 15:40–16:10

Stage 2

Lean Intelligence: Lessons from GitHub Copilot Data Science Efforts

Rahul Pandita

Researcher and Technical Advisor at Microsoft

Rahul Pandita
Open session

World Congress 2026 North America

September 24, 2026 · 17:30–18:00

Stage 6

No Single Model to Rule Them All: Building Resilient AI Agents Across Open & Closed LLMs

Emmanuel Acheampong

Senior Manager Developer Relations at Crusoe AI

Emmanuel Acheampong

Related videos

The Limits of Prompting: ArchitectingTrustworthy Coding Agents thumbnail
24:05
World Congress 2025 • about 1 year ago

The Limits of Prompting: ArchitectingTrustworthy Coding Agents

Nimrod Kor

How we built an AI-powered code reviewer in 80 hours thumbnail
28:07
World Congress 2025 • about 1 year ago

How we built an AI-powered code reviewer in 80 hours

Yan Cui

Teaching an LLM to review code … like a Senior Engineer! thumbnail
32:26
World Congress 2026 Europe • about 2 months ago

Teaching an LLM to review code … like a Senior Engineer!

Kesha Mykhailov

Developer Productivity Using AI Tools and Services - Ryan J Salva thumbnail
59:16
Coffee With Developers • over 1 year ago

Developer Productivity Using AI Tools and Services - Ryan J Salva

Ryan J Salva

Validating Real-World Usefulness of AI Models - Julia Kasper thumbnail
22:00
Coffee With Developers • 19 days ago

Validating Real-World Usefulness of AI Models - Julia Kasper

Julia Kasper

Beyond the IDE: A new era of agent collaboration thumbnail
27:12
World Congress 2025 • about 1 year ago

Beyond the IDE: A new era of agent collaboration

Ryan J. Salva

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.