> Markdown version of [/videos/1186-using-llms-in-your-product](https://www.wearedevelopers.com/videos/1186-using-llms-in-your-product). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Using LLMs in your Product

Integrating LLMs means shifting from rigid code to probabilistic workflows. Master structured prompt engineering, mitigate prompt injections, and leverage RAG to build secure, dynamic AI features.

- **Speakers:** [Daniel Töws](https://www.wearedevelopers.com/@daniel-tows)
- **Event:** World Congress 2024
- **Published:** August 29, 2024
- **Duration:** 31:12
- **URL:** https://www.wearedevelopers.com/videos/1186-using-llms-in-your-product

## Summary

Integrating Large Language Models (LLMs) into custom products requires shifting from formal programming to probabilistic workflows that rely heavily on natural language. Using the OpenAI API as a foundation, developers can build interactive tooling by defining clear system messages, tracking the full stateless chat history, and managing context window constraints. Because LLMs inherently operate by guessing the next token, structured prompt engineering becomes a critical tool to minimize chaotic outputs, reduce hallucinations, and maintain consistent user intent across diverse interactions.

Effective system prompts rely on specific configurations: defining a distinct persona, providing explicit reference data to anchor responses, and splitting complex requirements into sequential steps. However, consumer-facing LLMs introduce unique security risks, such as prompt injection, where users can manipulate the model into abandoning its primary instructions. Developers can deploy techniques like "post-prompting"—appending critical guardrail rules at the end of the context array—to force the model to prioritize system instructions over user input, though multi-layered validation checks are often still required in production.

To bridge the gap between static model training data and real-time product functionality, developers can utilize function calling to enable the model to request live context dynamically. Whether executing an SQL query, triggering an email notification, or pulling up-to-date pricing from an external service, function calls allow the LLM to pause text generation, fetch missing structured data, and seamlessly resume user interaction. For larger enterprise datasets that exceed standard API token limits, incorporating Retrieval-Augmented Generation (RAG) offers a highly scalable alternative to fine-tuning, transforming foundational models into specialized product engines.

**Keywords:** openai API integration, prompt engineering techniques, LLM context window management, system message configuration, prompt injection mitigation, conversational AI state management, LLM function calling capability, retrieval-augmented generation, stateless REST architecture, reducing AI hallucinations, tokenization constraints, post-prompting security guardrails, external API callbacks, JSON payload validation, probabilistic logic handling

## Chapters

1. **Three pillars of integrating large language models into products** (00:02) — Developers can leverage modern generative AI frameworks through direct API integrations, prompt design, and function callbacks.
1. **Utilizing chat completion requests and managing context length** (01:51) — Sending full chat histories to stateless APIs requires understanding token limits and prioritizing models based on dataset size and recency.
1. **Applying prompt engineering best practices for reliable model outputs** (08:30) — Providing precise instructions with contextual personas and product references reduces ambiguity and minimizes factual hallucinations.
1. **Designing system messages and mitigating prompt injection security risks** (13:37) — Establishing rigid system messages helps prevent malicious user inputs from hijacking the intended conversation scope.
1. **Enhancing language models with backend logic using function calls** (17:55) — Exposing custom callback functions allows text generation models to retrieve real-time data or trigger external system actions.
1. **Exploring retrieval augmented generation and advanced prompting techniques** (22:18) — Implementing retrieval augmented generation provides a mechanism to query massive localized datasets without fine-tuning underlying models.
1. **Addressing context limits inference costs and reliable json generation** (24:18) — Managing persistent connection lifecycles involves summarizing historical data logs and validating structured outputs using explicit tool boundaries.

## Related Moments

- [Building agentic workflows using prompt engineering and language models](https://www.wearedevelopers.com/videos/1266-navigating-the-ai-revolution-in-software-development) (from "Navigating the AI Revolution in Software Development")
- [Evolving traditional coding logic into LLM prompting](https://www.wearedevelopers.com/videos/920-wwc24-ankit-patel-unlocking-the-future-breakthrough-application-performance-and-capabilities-with-nvidia) (from "WWC24 - Ankit Patel - Unlocking the Future Breakthrough Application Performance and Capabilities with NVIDIA")
- [Enhancing conversational intent through modern large language models](https://www.wearedevelopers.com/videos/1641-hello-jarvis-building-voice-interfaces-for-your-llms) (from "Hello JARVIS - Building Voice Interfaces for Your LLMS")
- [Understanding language models and autonomous executing agents](https://www.wearedevelopers.com/videos/1725-wearedevelopers-live-build-a-multi-ai-agents-game-master-with-strands-our-weekly-web-finds) (from "WeAreDevelopers LIVE - Build a multi AI agents game master with Strands & our weekly web finds")
- [The shift from human prompt engineering to AI-generated prompts](https://www.wearedevelopers.com/videos/100255-design-patterns-for-ai-products-in-2026) (from "Design Patterns For AI Products in 2026")
- [Enhancing language models with retrieval-augmented generation](https://www.wearedevelopers.com/videos/844-enter-the-brave-new-world-of-genai-with-vector-search) (from "Enter the Brave New World of GenAI with Vector Search")

## Related Articles

- [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [WWC24 Talk - Scott Hanselman - AI: Superhero or Supervillain?](https://www.wearedevelopers.com/magazine/469-wwc24-talk-scott-hanselman-ai-superhero-or-supervillain)
- [Prompt Engineering is a Job of the Past](https://www.wearedevelopers.com/magazine/342-prompt-engineering-is-a-job-of-the-past)

## Related Jobs

- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/588393-machine-learning-engineer) at **Twilio**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1355348-machine-learning-engineer) at **TWILIO**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**
- [AI & Machine Learning Engineer (all genders)](https://www.wearedevelopers.com/jobs/48217-ai-machine-learning-engineer-all-genders) at **msg**
- [Staff, Machine Learning Engineer (L4)](https://www.wearedevelopers.com/jobs/ext/1202639-staff-machine-learning-engineer-l4) at **Twilio**