> Markdown version of [/videos/1231-lies-damned-lies-and-large-language-models](https://www.wearedevelopers.com/videos/1231-lies-damned-lies-and-large-language-models). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lies, Damned Lies and Large Language Models

Why do large language models confidently lie? Naive RAG pipelines often make factuality hallucinations worse. Discover how to systematically evaluate foundation models and implement adaptive retrieval.

- **Speakers:** Jodie Burchell
- **Event:** WeAreDevelopers LIVE
- **Published:** October 30, 2024
- **Duration:** 29:40
- **URL:** https://www.wearedevelopers.com/videos/1231-lies-damned-lies-and-large-language-models

## Summary

Large Language Models (LLMs) often generate fluent but factually incorrect outputs, a phenomenon rooted in their architecture and training data. As auto-regressive transformer models scale, they shift from learning basic syntax to compressing vast amounts of parametric knowledge drawn from web archives like Common Crawl. Because filtering petabytes of web data is inherently complex, models inevitably absorb and replicate misconceptions, outdated facts, and low-quality information directly from their training sets. To effectively deploy LLMs, developers must distinguish between two primary failure modes: faithfulness hallucinations, where a model deviates from explicitly provided context, and factuality hallucinations, where a model confidently outputs incorrect parametric knowledge. Assessing a model's true capability requires testing against specialized benchmark datasets like TruthfulQA, which specifically targets common misconceptions and conspiracy theories. Utilizing frameworks like LangChain alongside datasets hosted by Hugging Face allows engineering teams to systematically evaluate hallucination rates via API integrations before committing to a specific foundation model. While minimizing temperature settings is a standard baseline tactic, more robust mitigation requires multi-layered strategies spanning from rigid prompt engineering and domain-specific fine-tuning to collaborative self-refinement. Retrieval-augmented generation (RAG) is the most prominent approach for injecting external, verified context into an LLM's prompt. However, implementing RAG introduces complex information retrieval hurdles involving chunking, embeddings, and vector databases. Naive RAG implementations can actually exacerbate factuality errors if irrelevant context is retrieved or if the chosen LLM suffers from high faithfulness hallucinations, meaning it cannot properly utilize the provided text. Adopting advanced adaptive techniques like retrieve only when needed (ROWEN) helps balance computational cost and accuracy by selectively applying RAG only when the probability of a baseline factuality hallucination is strictly high.

**Keywords:** LLM hallucinations, auto-regressive models, transformer architecture, parametric knowledge, common crawl filtering, faithfulness hallucinations, factuality hallucinations, truthful QA benchmark, langchain API integration, hugging face datasets, zero temperature prompting, large language model fine-tuning, retrieval-augmented generation, RAG pipeline challenges, document embeddings, vector databases, retrieve only when needed

## Chapters

1. **Introduction to large language model hallucinations** (00:00) — Autoregressive text generation models produce convincing but functionally incorrect outputs due to underlying dataset abstractions.
1. **Transformer architecture and training data scalability** (02:25) — Decoder-based algorithms scale continuously by using next-word prediction across massive datasets to improve sequence generation capabilities.
1. **Evolution of GPT models and lossy compression** (04:01) — Increasing parameters in successive model generations enables the retention of structural grammar alongside specific parametric knowledge.
1. **Filtering unstructured web data for model training** (06:55) — Filtering offensive or duplicate content from datasets like Common Crawl severely impacts generated model output quality.
1. **Distinguishing faithfulness and factuality hallucination errors** (09:45) — Models exhibit errors by either ignoring explicit prompt context or relying on incorrect training data.
1. **Measuring text hallucinations with standardized evaluation datasets** (11:15) — Identifying factual inaccuracies and ingrained misconceptions requires standardized benchmark frameworks like TruthfulQA and Squad.
1. **Demonstrating an open source TruthfulQA evaluation pipeline** (13:48) — Testing multiple-choice misconception queries against language models requires setting up benchmarks using LangChain and Hugging Face.
1. **Analyzing large language model benchmark evaluation results** (19:37) — Interpreting factuality error rates from testing frameworks enables leveraging community-driven leaderboards instead of manual validation.
1. **Techniques for mitigating model hallucination output frequencies** (21:26) — Mitigation strategies include context-heavy constraint prompting, domain-specific fine-tuning, and algorithmic output evaluation techniques used alongside retrieval augmentation.
1. **Structuring effective retrieval augmented generation system architectures** (24:02) — Integrating dynamic vector database bounds into text inference adds targeted context while introducing potential data retrieval inaccuracies.
1. **Implementing advanced context patterns to reduce hallucinations** (28:09) — Using conditional retrieval methodologies like ROWAN limits unnecessary vector queries when baseline model confidence is high.

## Related Moments

- [Fundamentals and limitations of large language models](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) (from "Innovating Developer Tools with AI: Insights from GitHub Next")
- [Overcoming language model challenges using retrieval-augmented generation](https://www.wearedevelopers.com/videos/1249-building-blocks-of-rag-from-understanding-to-implementation) (from "Building Blocks of RAG: From Understanding to Implementation")
- [Mitigating artificial intelligence hallucinations with constraints and context](https://www.wearedevelopers.com/videos/994-mastering-ai-driven-problem-solving-in-engineering-with-observability) (from "Mastering AI-Driven Problem Solving in Engineering with Observability")
- [Overcoming hallucination challenges in large language models](https://www.wearedevelopers.com/videos/1154-large-language-models-knowledge-graphs) (from "Large Language Models ❤️ Knowledge Graphs")
- [Addressing the risks of hallucination and chat model manipulation](https://www.wearedevelopers.com/videos/1093-ai-is-dead-long-live-ak) (from "AI is dead, long live AK")
- [Hallucinations and factual inaccuracies in generative language models](https://www.wearedevelopers.com/videos/1323-martin-o-hanlon-make-llms-make-sense-with-graphrag) (from "Martin O'Hanlon - Make LLMs make sense with GraphRAG")

## Related Articles

- [ChatGPT on AI Hallucinations: Can It Fix Its Own Mistakes?](https://www.wearedevelopers.com/magazine/566-chatgpt-on-ai-hallucinations-can-it-fix-its-own-mistakes)
- [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models)
- [The Best Large Language Models on The Market](https://www.wearedevelopers.com/magazine/319-the-best-large-language-models-on-the-market)
- [How machine learning can help us tell fact from fiction](https://www.wearedevelopers.com/magazine/509-how-machine-learning-can-help-us-tell-fact-from-fiction)

## Related Jobs

- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/588393-machine-learning-engineer) at **Twilio**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1355348-machine-learning-engineer) at **TWILIO**
- [Staff, Machine Learning Engineer (L4)](https://www.wearedevelopers.com/jobs/ext/1202639-staff-machine-learning-engineer-l4) at **Twilio**
- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**
- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**