Stop Guessing, Start Measuring: Evaluating RAG Systems with Synthetic Test Data

Most RAG systems go to production without proper evaluation — mainly because building a comprehensive test dataset feels like a project in itself. What if it doesn’t have to be? In this live coding session, we’ll generate a synthetic test dataset using RAGAS and its knowledge graph approach — no manual question-answer curation needed. Using the public dataset as our document base, we’ll build a test dataset from scratch and run it against an existing RAG pipeline, all live. Along the way, we’ll measure the metrics that actually tell you whether your system works: context precision, context recall, noise sensitivity, response relevancy, and faithfulness. You’ll see how synthetic test data from knowledge graphs covers edge cases that manually curated sets typically miss. This session comes from production experience with RAGAS and is designed for developers who work with RAG, LLMs and AI agents, and want a reliable, low-effort evaluation workflow they can take home and use immediately.