> Markdown version of [/videos/100319-llms-in-the-wild-building-an-ai-agent-that-survives-production?t=568](https://www.wearedevelopers.com/videos/100319-llms-in-the-wild-building-an-ai-agent-that-survives-production?t=568). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# LLMs in the wild: Building an AI agent that survives production

Stop letting prompt tweaks break your production AI. Learn how GetYourGuide used test-driven development and rigorous CI/CD evaluations to safely deploy over 150 prompt modifications.

- **Speakers:** [Steven Mi](https://www.wearedevelopers.com/@steven-mi), [Giampaolo Casolla](https://www.wearedevelopers.com/@giampaolo-casolla)
- **Event:** World Congress 2026 Europe
- **Published:** July 10, 2026
- **Duration:** 28:06
- **URL:** https://www.wearedevelopers.com/videos/100319-llms-in-the-wild-building-an-ai-agent-that-survives-production

## Summary

Keyword search struggles with the open-ended, nuanced requests typical of travel discovery. To address this, GetYourGuide moved beyond basic ChatGPT wrappers—which suffer from slow tool calls and data leakage—to build a specialized agentic workflow. This custom orchestration layer transforms natural language constraints into structured queries, unlocking deeper and highly relevant travel recommendations without trapping users in a lagging agentic loop.

The resulting architecture routes prompts through multiple targeted nodes: an intent extractor compresses conversational history, a planner structuralizes fields, and a drafter composes the final response. Under the hood, the system integrates hybrid BM25 and semantic search combined with a deep learning ranking engine. A dedicated large language model acts as a relevance filter on the top retrieved candidates to discard superficial matches. For scale and resilience, the system employs fan-out execution with latency cutoffs and dynamic model routing, dispatching simpler queries to lighter, more affordable models.

Moving from prototype to a resilient production system highlighted that "all LLM projects are easy until they aren't," as prompt tweaks frequently trigger whack-a-mole regressions. To confidently iterate, the team adopted test-driven development for AI, building a golden dataset of over 800 handcrafted test cases evaluated within their CI/CD pipeline using DeepEval and LangGraph. Relying heavily on deterministic evaluations and pass-at-k thresholds to handle non-determinism enabled them to deploy over 150 prompt modifications with zero code-change regressions, proving that LLM systems must be treated as robust production code rather than a demo that just got promoted.

**Keywords:** LLM production deployment, conversational ai discovery, agentic workflow orchestration, hybrid search retrieval, semantic search filtering, prompt regression testing, ai evaluation frameworks, test-driven ai development, CI/CD for LLMs, dynamic model routing, deterministic ai evaluation, pass-at-k metric, LLM wrapper limitations, langgraph orchestration, deepeval integration

## Chapters

1. **Navigating the gap between prototype and production** (00:36) — Exploring the common challenge of migrating language model prompts to predictable behaviors in production systems.
1. **Motivation for introducing a conversational discovery agent** (03:28) — How changes in the search landscape and user preferences drove the need for automated travel discovery.
1. **Extracting complex semantic intent from user queries** (05:47) — Transforming unstructured and highly constrained travel plans into parseable criteria for backend services.
1. **Moving from API wrappers to agentic orchestration** (07:19) — Orchestrating states locally to improve performance and prevent data leakage observed in platform API wrappers.
1. **Architecting the multi-step travel discovery pipeline** (09:28) — Breaking the discovery journey into specific intent, planning, retrieval, relevance, and drafting components.
1. **Optimizing inference latency and node failure resilience** (14:35) — Implementing fan-out parallelization, fallback paths, and varying model complexities to reduce processing overhead.
1. **Preventing feature regression using test-driven development** (16:20) — How continuous prompt iterations lead to edge case failures without a rigorous evaluation framework.
1. **Designing concrete metrics for deterministic component evaluations** (19:36) — Crafting handcrafted input targets specifying expected fields, explicit null values, and routing logic flags.
1. **Automating regression checks in continuous integration loops** (21:53) — Triggering automated tests with a pass-at-k strategy to confidently merge iterations of variable models.
1. **Key takeaways for reliable production agent integrations** (25:09) — Integrating rigorous deterministic tests from inception and prioritizing system-grade development for feature deployments.
1. **Tooling and language coverage in question responses** (26:49) — Discussing the application of specific observation frameworks and processing inputs in numerous global languages.

## Related Moments

- [Introducing LLMs as judges for automated testing](https://www.wearedevelopers.com/videos/100300-testing-ai-agents-automated-evaluation-for-chatbots-rag-systems) (from "Testing AI Agents: Automated Evaluation for Chatbots & RAG Systems")
- [Transitioning from AI co-pilots to AI-native products](https://www.wearedevelopers.com/videos/100091-3-ways-to-rebuild-the-data-stack-for-agents) (from "3 Ways to Rebuild the Data Stack for Agents")
- [Enhancing conversational intent through modern large language models](https://www.wearedevelopers.com/videos/1641-hello-jarvis-building-voice-interfaces-for-your-llms) (from "Hello JARVIS - Building Voice Interfaces for Your LLMS")
- [Differentiating AI search layers from standard LLMs](https://www.wearedevelopers.com/videos/100133-ai-search-insights-from-otterlyai-what-we-tested-what-failed-and-what-actually-works) (from "AI Search Insights from OtterlyAI: What We Tested, What Failed, and What Actually Works")
- [Building components of a real-world LLM lifecycle](https://www.wearedevelopers.com/videos/1250-from-traction-to-production-maturing-your-llmops-step-by-step) (from "From Traction to Production: Maturing your LLMOps step by step")
- [Leveraging generative AI and agents for executive productivity](https://www.wearedevelopers.com/videos/1360-inside-mercedes-benz-how-cio-katrin-lehmann-is-empowering-5-000-developers-and-driving-digital-change) (from "Inside Mercedes-Benz: How CIO Katrin Lehmann is Empowering 5,000 Developers and Driving Digital Change")

## Related Articles

- [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models)
- [From Prototype to Production: Build AI Agents with This Free 4-Course Learning Path](https://www.wearedevelopers.com/magazine/655-from-prototype-to-production-build-ai-agents-with-this-free-4-course-learning-path)
- [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering)
- [Never delegate the understanding](https://www.wearedevelopers.com/magazine/749-never-delegate-the-understanding)

## Related Jobs

- [Senior AI Agent Software Engineer (Go, Python) (m/f/x)](https://www.wearedevelopers.com/jobs/48277-senior-ai-agent-software-engineer-go-python-m-f-x) at **Dynatrace**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/588393-machine-learning-engineer) at **Twilio**
- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1355348-machine-learning-engineer) at **TWILIO**
- [AI Operations Manager (all genders)](https://www.wearedevelopers.com/jobs/48263-ai-operations-manager-all-genders) at **envelio**