> Markdown version of [/videos/100324-teaching-an-llm-to-review-code-like-a-senior-engineer](https://www.wearedevelopers.com/videos/100324-teaching-an-llm-to-review-code-like-a-senior-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Teaching an LLM to review code … like a Senior Engineer!

Are out-of-the-box LLMs sabotaging your pull requests with noisy nitpicks? Discover how one team built an evaluation flywheel to train an AI that reviews code like a senior engineer.

- **Speakers:** [Kesha Mykhailov](https://www.wearedevelopers.com/@kesha-mykhailov)
- **Event:** World Congress 2026 Europe
- **Published:** July 10, 2026
- **Duration:** 32:26
- **URL:** https://www.wearedevelopers.com/videos/100324-teaching-an-llm-to-review-code-like-a-senior-engineer

## Summary

Building an LLM code review bot that mimics a senior engineer reveals that helpful AI defaults often sabotage developer productivity. When applied naively to pull requests, LLMs produce lavish, overwhelmingly noisy reviews filled with irrelevant nitpicks that engineers quickly ignore. Initial attempts to use out-of-the-box LLM agents resulted in massive AI-generated slop, proving that simply capturing expert domain knowledge into markdown guidelines is not enough to maximize the signal-to-noise ratio.

To solve this, the engineering team adopted a rigorous evaluation flywheel, treating the AI agent like any mission-critical production system. They harvested historical GitHub reviews to establish a ground truth and built a mock PR review ecosystem using Docker containers alongside an LLM judge. By leveraging batch evaluations and offline AB testing, they fine-tuned system prompts and isolated false positives down to individual word choices. This tight feedback loop—measuring direct user validation and indirect applied-commit signals—ultimately pushed the agent's effectiveness past a critical 50% acceptance threshold.

Beyond surfacing defects, the ultimate goal was achieving safe, automated PR approvals. Introducing a deterministic line-of-code limit forced developers to split massive, risky AI-generated diffs into concise chunks, deeply reinforcing a culture of shipping small and fast. Naming the rigorous bot "Shrek" deliberately cut through widespread AI fatigue by giving the tool a memorable persona. This sociotechnical system created clear behavioral incentives: engineers modularized their code to earn instant approvals, and actively documented their architectural decisions to push back on the agent's findings, inadvertently improving long-term institutional knowledge sharing.

**Keywords:** LLM code review bot, PR automation, AI agent evaluation flywheel, mock MCP server, LLM judge framework, batch evaluations, offline AB testing, false positive suppression, deterministic code gates, safe shipping culture, sociotechnical system design, ground truth curation, developer productivity AI, AI-generated code challenges, automated PR approval

## Chapters

1. **Building reliable AI agents for business value** (01:25) — Demonstrable success in automated customer resolution proves that highly effective agents require relentless experimentation and custom models.
1. **Why generic LLMs fail at code reviews** (04:42) — Standard language models produce verbose and irrelevant code feedback because they lack foundational knowledge about unique organizational frameworks.
1. **Leveraging historical context to customize code agents** (07:19) — Automating pull request comments using raw historical design guidelines leads to overwhelming noise without proper deployment testing.
1. **Building an offline evaluation framework for code reviews** (09:15) — Running simulated evaluations inside isolated containers captures structural issues before untested bots can pollute live repositories.
1. **Refining AI prompts through batch evaluations** (10:46) — Using an automated judge to measure severity and correctness helps eliminate false positives triggered by overly rigid prompt formatting.
1. **Gathering human signals to validate agent helpfulness** (13:04) — Mining direct interface feedback and implied commit changes reveals true metrics for an automated reviewer's operational value.
1. **Expanding AI agents to handle pull request approvals** (14:42) — Deconstructing the approval process assigns distinct tasks to specialized sub-agents for catching fundamental bugs and architectural violations.
1. **Designing agent-first interfaces for developer workflows** (16:48) — Constructing specialized interaction protocols ensures that automated assistants can curate evaluation datasets without disrupting natural programming habits.
1. **Preventing deployment risks using deterministic commit limits** (17:59) — Limiting automatic approvals to exceptionally short architectural changes forces developers to embrace continuous delivery and mitigate production flaws.
1. **Shaping sociotechnical systems with character-driven automation** (21:05) — Injecting distinguished personas into pipeline tooling cuts through alert fatigue and naturally encourages engineers to divide dense assignments.
1. **Managing unexpected consequences in automated workflow gating** (24:38) — Intense mechanical bottlenecks frequently cause frustration but inadvertently push engineers to preserve intricate logic rationale inside version histories.
1. **Strategic requirements for implementing enterprise AI agents** (26:21) — Successfully scaling an intelligent code assistant demands continuous background iteration frameworks alongside explicit social reinforcement techniques.
1. **Retaining knowledge sharing inside generative coding environments** (28:50) — Tracking granular design choices during automated coding sessions enables experienced developers to mentor juniors asynchronously.

## Related Moments

- [Reducing pull request cycle times with artificial intelligence](https://www.wearedevelopers.com/videos/1698-state-of-devex-2025-navigating-the-ai-revolution-and-developer-experience) (from "State of DevEx 2025: Navigating the AI Revolution and Developer Experience")
- [Enforcing automated code reviews to manage accelerated delivery](https://www.wearedevelopers.com/videos/1902-behind-the-scenes-of-building-vs-code-harald-kirschner) (from "Behind the Scenes of Building VS Code - Harald Kirschner")
- [Replacing traditional code reviews with interactive agents](https://www.wearedevelopers.com/videos/100353-meta-harnesses-what-the-js-framework-wars-teach-us-about-what-is-next-for-ai-agents) (from "Meta Harnesses: What the JS Framework Wars Teach Us About What Is Next for AI Agents")
- [Introducing LLMs as judges for automated testing](https://www.wearedevelopers.com/videos/100300-testing-ai-agents-automated-evaluation-for-chatbots-rag-systems) (from "Testing AI Agents: Automated Evaluation for Chatbots & RAG Systems")
- [Automating pull request feedback with AI code review agents](https://www.wearedevelopers.com/videos/1462-evaluating-ai-models-for-code-comprehension) (from " Evaluating AI models for code comprehension")
- [Shifting developer workloads and realistic AI productivity gains](https://www.wearedevelopers.com/videos/1830-wearedevelopers-live-speculaitions) (from "WeAreDevelopers LIVE - SpeculAItions")

## Related Articles

- [Never delegate the understanding](https://www.wearedevelopers.com/magazine/749-never-delegate-the-understanding)
- [How to Use Generative AI to Accelerate Learning to Code](https://www.wearedevelopers.com/magazine/530-how-to-use-generative-ai-to-accelerate-learning-to-code)
- [AI-Powered Debugging: The Future of Fixing Your Code](https://www.wearedevelopers.com/magazine/553-ai-powered-debugging-the-future-of-fixing-your-code)
- [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models)

## Related Jobs

- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**
- [Senior AI Agent Software Engineer (Go, Python) (m/f/x)](https://www.wearedevelopers.com/jobs/48277-senior-ai-agent-software-engineer-go-python-m-f-x) at **Dynatrace**
- [Principal Software Engineer, Enterprise AI Platform](https://www.wearedevelopers.com/jobs/ext/1467292-principal-software-engineer-enterprise-ai-platform) at **GitHub**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/588393-machine-learning-engineer) at **Twilio**
- [Principal Product Manager, Agent Platform](https://www.wearedevelopers.com/jobs/ext/277541-principal-product-manager-agent-platform) at **GitHub**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1355348-machine-learning-engineer) at **TWILIO**