WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe - Virtual Stage • Jul 2, 2026 • Session details

Stop Guessing, Start Measuring: Evaluating RAG Systems with Synthetic Test Data

Csenge Szabo

Are your RAG applications failing silently? Stop relying on user complaints to catch hallucinations. Learn to independently measure retrieval and generation using RAGAS and synthetic data.

Pause
Mute Enter Fullscreen
#1 about 3 min

Establishing rigorous evaluation pipelines for operational RAG systems

Preventing hallucinated answers and poor data grounding requires setting up comprehensive observability pipelines before application deployment.

#2 about 2 min

Understanding overarching retrieval and generation steps in RAG architectures

Connecting source documents into vector stores supports systemic context retrieval and dynamic model answer generation.

#3 about 2 min

Isolating independent failure surfaces in RAG application pipelines

Diagnosing silent breakages properly involves separating the underlying accuracy of retrieval logic from final component generation.

#4 about 3 min

Overcoming volume and coverage limits in evaluation datasets

Handcrafting exact gold labels manually proves too slow and expensive to accommodate adequate complexity and request variety.

#5 about 4 min

Generating synthetic test data using structured open frameworks

Bypassing generic model traversals in favor of graph-structured tools prevents shallow and overly repetitious question extraction.

#6 about 2 min

Defining varied query types via knowledge graph topologies

Distinguishing between multi-hop and single-hop abstraction ensures evaluation regimes effectively challenge complex chunk retrieval mechanics.

#7 about 3 min

Synthesizing customized test sets from interconnected node relationships

Drawing concrete connections between text snippets through shared themes empowers highly realistic interactive persona emulation.

#8 about 7 min

Automating synthetic test set compilation via Python scripts

Leveraging framework integrations splits raw markdown documentation into optimally overlapping chunks for automated JSON graph formatting.

#9 about 6 min

Diagnosing retrieval and generation pipelines with quantifiable metrics

Measuring context precision alongside noise sensitivity precisely isolates flawed ranking algorithms from poor system prompting.

#10 about 3 min

Executing automated evaluation suites leveraging LLMs as judges

Scoring synthetic dataset queries programmatically against system responses provides an objective baseline for fine-tuning configuration changes.

#11 about 4 min

Incorporating human validation to mitigate automated model bias

Countering inherent system bias requires evaluating synthetically labeled test datasets alongside genuine faults identified in production logs.

#12 about 2 min

Establishing baseline observability and evaluation prior to launch

Adopting quantifiable data metrics from day one proactively prevents system blind spots across evolving architectural revisions.

Matching moments

38 sec

Introducing OpenRAG for custom data pipelines

Phil Nash · Coffee With Developers

1:34 min

Mitigating the inherent challenges of generative AI tools

Mary Grygleski Mary Grygleski · LIVE

3:22 min

Evaluating advanced artificial intelligence platforms for daily recruitment

Rudi Bauer Rudi Bauer +1 · Cappuccino with HR

2:12 min

Deploying conversational intelligence tools for complex unstructured data

Niklas Blumenthal Niklas Blumenthal +1 · World Congress 2025

1:50 min

Simplifying generative AI deployments using the RagStack opinionated framework

David Leconte David Leconte +1 · World Congress 2024

1:44 min

Understanding basic retrieval-augmented generation architectures in chatbots

Stan Girard Stan Girard · World Congress 2024

More from World Congress 2026 Europe - Virtual Stage

One Does Not Simply Make Architecture Decisions Alone thumbnail
36:15
about 2 months ago

One Does Not Simply Make Architecture Decisions Alone

Vladas Diržys

From Black Box to Glass Box : Bedrock AgentCore Observability thumbnail
21:15
2 months ago

From Black Box to Glass Box : Bedrock AgentCore Observability

Yasemin Aktürk

Leading in Rhythm: How Women Make Better Decisions with Biological Intelligence thumbnail
30:41
2 months ago

Leading in Rhythm: How Women Make Better Decisions with Biological Intelligence

Julia Grosse

Better Together: Leveraging Your Observability Tools as a SIEM thumbnail
39:06
2 months ago

Better Together: Leveraging Your Observability Tools as a SIEM

Mathias Palmersheim

ChatGPT Apps: From Chat to Checkout thumbnail
30:29
2 months ago

ChatGPT Apps: From Chat to Checkout

Aaron Koivunen

Interior Decorating (TypeScript Decorators) thumbnail
29:22
2 months ago

Interior Decorating (TypeScript Decorators)

Michael Dowden

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 24, 2026 · 11:40–12:10

Stage 4

Taming Rogue Agents: Observability-Driven Evaluation for Production Reliability

Anagha Rumade, Anjana Umapathy, Apoorva Jaiswal

Anagha Rumade
Anjana Umapathy
Apoorva Jaiswal
Open session

World Congress 2026 North America

September 24, 2026 · 16:50–17:20

Stage 6

Who Tests the AI? Building Trustworthy AI Systems at Enterprise Scale

Him Raj Singh

PayPal, Manager, Software Engineer

Him Raj Singh
Open session

World Congress 2026 North America

September 24, 2026 · 11:20–11:25

Outdoor Stage

Finding the Edges: Testing, Evaluating, and Monitoring Voice AI Agents Before Your Users Do

Matt Wyman

CEO at Okareo

Matt Wyman
Open session

World Congress 2026 North America

September 25, 2026 · 15:30–16:00

Stage 4

Evals Are Infra: Building AI Systems Developers Can Actually Trust

Phoebe Wang

Member of Technical Staff at OpenAI

Phoebe Wang
Open session

World Congress 2026 North America

September 25, 2026 · 16:50–17:20

Stage 3

Your Evals Passed. Your Agent Just Emptied a Database.

Tejas Pravinbhai Patel

IEEE Award-Winning Researcher | Best Keynote Speaker | Sr. Software Engineer at Amazon | AI Systems & Agent Architect

Tejas Pravinbhai Patel
Open session

World Congress 2026 North America

September 25, 2026 · 11:40–12:10

Stage 2

Reinventing Testing Practices in the AI Era

Eric Deandrea

Java Champion & Senior Principal Software Engineer, IBM

Eric Deandrea

Related videos

Building Blocks of RAG: From Understanding to Implementation thumbnail
26:25
LIVE • almost 2 years ago

Building Blocks of RAG: From Understanding to Implementation

Ashish Sharma

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation thumbnail
21:17
World Congress 2024 • almost 2 years ago

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation

Carl Lapierre

Truth, Lies, and Probabilities: Testing AI Hallucinations thumbnail
32:14
World Congress 2026 Europe • about 2 months ago

Truth, Lies, and Probabilities: Testing AI Hallucinations

Anastasia Simou

Testing AI Agents: Automated Evaluation for Chatbots & RAG Systems thumbnail
30:34
World Congress 2026 Europe • about 2 months ago

Testing AI Agents: Automated Evaluation for Chatbots & RAG Systems

Sebastian Messingfeld

Lies, Damned Lies and Large Language Models thumbnail
29:40
LIVE • almost 2 years ago

Lies, Damned Lies and Large Language Models

Jodie Burchell

Scaling GraphRAG: Efficient Knowledge Retrieval for AI thumbnail
24:04
World Congress 2026 Europe • about 2 months ago

Scaling GraphRAG: Efficient Knowledge Retrieval for AI

Gal Shubeli

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.