Max Tkacz

The AI Agent Path to Prod: Building for Reliability

Your AI agent works in demos, but will it break in production? Learn to build the evaluation frameworks and guardrails necessary for true reliability.

The AI Agent Path to Prod: Building for Reliability
#1about 4 minutes

Why AI agents fail in production environments

AI agents often fail in production because the probabilistic nature of LLMs conflicts with the need for reliability at scale.

#2about 5 minutes

Scoping an AI agent for a specific business problem

Start by identifying a low-risk, high-impact task, like automating free trial extensions, to establish a viable solution scope.

#3about 3 minutes

Walking through the naive V1 customer support agent

The initial agent uses an LLM with tools to fetch user data and extend trials, but its reliability is unknown without testing.

#4about 4 minutes

Using evaluations to test the happy path case

Evaluations are introduced as a testing framework to run the agent against specific test cases, revealing inconsistencies even in the happy path.

#5about 4 minutes

Improving agent consistency with prompt engineering

By adding explicit rules and few-shot examples to the system prompt, the agent's tool usage and response quality become more consistent.

#6about 5 minutes

Testing for prompt injection and other edge cases

A new evaluation case for prompt injection reveals a security flaw, which is fixed by adding specific security rules to the system prompt.

#7about 6 minutes

Applying production guardrails beyond evaluations

Beyond evals, production readiness requires adding human-in-the-loop processes, custom error handling, rate limiting, and model redundancy.

Related jobs
Jobs that call for the skills explored in this talk.

Featured Partners

Related Videos

Agents for the Sake of Happiness39:11

Agents for the Sake of Happiness

Thomas Dohmke

Beyond Chatbots: How to build Agentic AI systems24:22

Beyond Chatbots: How to build Agentic AI systems

Philipp Schmid

On a Secret Mission: Developing AI Agents29:03

On a Secret Mission: Developing AI Agents

Jörg Neumann

Beyond Prompting: Building Scalable AI with Multi-Agent Systems and MCP25:32

Beyond Prompting: Building Scalable AI with Multi-Agent Systems and MCP

Viktoria Semaan

The Limits of Prompting: ArchitectingTrustworthy Coding Agents24:05

The Limits of Prompting: ArchitectingTrustworthy Coding Agents

Nimrod Kor

The State of GenAI & Machine Learning in 202526:47

The State of GenAI & Machine Learning in 2025

Alejandro Saucedo

Three years of putting LLMs into Software - Lessons learned26:30

Three years of putting LLMs into Software - Lessons learned

Simon A.T. Jiménez

You are not my model anymore - understanding LLM model behavior24:11

You are not my model anymore - understanding LLM model behavior

Andreas Erben

From learning to earning

Jobs that call for the skills explored in this talk.