Nimrod Kor

The Limits of Prompting: ArchitectingTrustworthy Coding Agents

Prompt engineering has its limits. Learn how a multi-agent architecture, enriched with deep context, boosted our AI agent's suggestion acceptance rate from 12% to over 60%.

The Limits of Prompting: ArchitectingTrustworthy Coding Agents
#1about 2 minutes

Prototyping a basic AI code review agent

A simple prototype using a GitHub webhook and a single LLM call reveals the potential for understanding code semantics beyond static analysis.

#2about 2 minutes

Iteratively improving prompts to handle edge cases

Simple prompts fail to consider developer comments or model knowledge cutoffs, requiring more detailed instructions to improve accuracy.

#3about 5 minutes

Establishing a robust benchmarking process for agents

A reliable benchmarking pipeline uses a large dataset, concurrent execution, and an LLM-as-a-judge (LLJ) to measure and track performance improvements.

#4about 2 minutes

Decomposing large tasks into specialized agents

To combat inconsistency and hallucinations, a single large task like code review is broken down into multiple smaller, specialized agents.

#5about 6 minutes

Leveraging codebase context for deeper insights

Moving beyond prompts, providing codebase context via vector similarity (RAG) and module dependency graphs (AST) unlocks high-quality, human-like feedback.

#6about 3 minutes

Introducing Awesome Reviewers for community standards

Awesome Reviewers is a collection of prompts derived from open-source projects that can be used to enforce team-specific coding standards.

#7about 1 minute

Key takeaways for building reliable LLM agents

The path to a reliable agent involves starting with a proof-of-concept, benchmarking rigorously, using prompt engineering for quick fixes, and investing in deep context.

Related jobs
Jobs that call for the skills explored in this talk.

Featured Partners

Related Videos

How we built an AI-powered code reviewer in 80 hours28:07

How we built an AI-powered code reviewer in 80 hours

Yan Cui

Three years of putting LLMs into Software - Lessons learned26:30

Three years of putting LLMs into Software - Lessons learned

Simon A.T. Jiménez

The AI Agent Path to Prod: Building for Reliability30:48

The AI Agent Path to Prod: Building for Reliability

Max Tkacz

Prompt Engineering - an Art, a Science, or your next Job Title?33:28

Prompt Engineering - an Art, a Science, or your next Job Title?

Maxim Salnikov

Bringing the power of AI to your application.32:26

Bringing the power of AI to your application.

Krzysztof Cieślak

Beyond Prompting: Building Scalable AI with Multi-Agent Systems and MCP25:32

Beyond Prompting: Building Scalable AI with Multi-Agent Systems and MCP

Viktoria Semaan

AI: Superhero or Supervillain? How and Why with Scott Hanselman25:17

AI: Superhero or Supervillain? How and Why with Scott Hanselman

Scott Hanselman

Using LLMs in your Product31:12

Using LLMs in your Product

Daniel Töws

From learning to earning

Jobs that call for the skills explored in this talk.