> Markdown version of [/playlists/llm-evaluation](https://www.wearedevelopers.com/playlists/llm-evaluation). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Playlist: LLM evaluation 12 videos · 13 moments · 45.1 minutes ## Evals vs. Evil - AI and Package Security - Laurie Voss - **Defining and implementing LLM evaluation strategies** (00:02, 2min) — Because traditional unit tests fail on variable LLM outputs, utilizing a separate model for evaluation serves as an effective testing substitute. [Learn more](https://www.wearedevelopers.com/videos/2131-evals-vs-evil-ai-and-package-security-laurie-voss) ## Lessons Learned Building a GenAI Powered App - **Evaluating LLM response accuracy using secondary LLMs** (25:32, 2min) — Creating an automated validation loop where a secondary model grades the initial facts. [Learn more](https://www.wearedevelopers.com/videos/1156-lessons-learned-building-a-genai-powered-app) ## LLMOps-driven fine-tuning, evaluation, and inference with NVIDIA NIM & NeMo Microservices - **Executing fine-tuning and LLM evaluation API workflows** (12:29, 4min) — Pulling foundation models and datasets via API requests enables automated container training and adapter benchmarking. [Learn more](https://www.wearedevelopers.com/videos/1582-llmops-driven-fine-tuning-evaluation-and-inference-with-nvidia-nim-nemo-microservices) ## AI as a Test Designer: Transforming Experience into Automated Testing - **Validating test cases using the LLM judge concept** (17:47, 8min) — Deploying a secondary LLM model evaluates newly generated test scenarios for quality and relevance against baseline usage patterns. [Learn more](https://www.wearedevelopers.com/videos/1984-ai-as-a-test-designer-transforming-experience-into-automated-testing) ## Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source - **Addressing system transparency when utilizing an LLM judge** (27:51, 1min) — Leveraging deep evaluation traces clarifies the logical steps and exact reasoning behind individual automated assessment scores. - **Programmatic model evaluation and custom metrics via MLflow** (10:14, 4min) — Using an LLM as a judge alongside clear guidelines within MLflow creates an automated baseline for continuous improvement. [Learn more](https://www.wearedevelopers.com/videos/100151-cutting-llm-costs-without-cutting-quality-how-to-beat-proprietary-llms-with-fine-tuned-open-source) ## Testing AI Agents: Automated Evaluation for Chatbots & RAG Systems - **Introducing LLMs as judges for automated testing** (12:29, 3min) — Using independent language models to critically review chatbot outputs at scale for correctness and consistency. [Learn more](https://www.wearedevelopers.com/videos/100300-testing-ai-agents-automated-evaluation-for-chatbots-rag-systems) ## MAD About Software Design - When AI Architects Argue - **Evaluating AI debate quality against formal architecture katas** (16:14, 3min) — Systematic testing reveals that clarifications improve results while summarization and excessive debate rounds degrade design quality. [Learn more](https://www.wearedevelopers.com/videos/1952-mad-about-software-design-when-ai-architects-argue) ## Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models - **Evaluating prompt accuracy against expected test outcomes** (08:00, 2min) — Establishing test datasets and using LLM judges to ensure accuracy and coherence. [Learn more](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models) ## Give Your LLMs a Left Brain - **Challenges of adopting LLMs for enterprise applications** (00:02, 3min) — Commercial LLMs trained on broad datasets face significant obstacles when applied to specific enterprise contexts and problems. [Learn more](https://www.wearedevelopers.com/videos/1160-give-your-llms-a-left-brain) ## Stop Guessing, Start Measuring: Evaluating RAG Systems with Synthetic Test Data - **Executing automated evaluation suites leveraging LLMs as judges** (26:44, 2min) — Scoring synthetic dataset queries programmatically against system responses provides an objective baseline for fine-tuning configuration changes. [Learn more](https://www.wearedevelopers.com/videos/1982-stop-guessing-start-measuring-evaluating-rag-systems-with-synthetic-test-data) ## 11 Principles for Evaluating AI Dev Tools - **Separating autonomous code generation from verification concerns** (15:25, 1min) — Reducing cognitive bias in LLM outputs requires independent evaluation harnesses rather than letting agents review themselves. [Learn more](https://www.wearedevelopers.com/videos/100307-11-principles-for-evaluating-ai-dev-tools) ## Hack Me If You Can: Designing Unbreakable LLM Guardrails - **Implementing regex, classifiers, and LLM-as-a-judge guardrails** (13:36, 3min) — Teams construct layered defenses by mixing cheap regex rules with lightweight structural classifiers and analytical evaluation models. [Learn more](https://www.wearedevelopers.com/videos/100209-hack-me-if-you-can-designing-unbreakable-llm-guardrails)