WEBVTT

1
00:00:02.680 --> 00:00:06.390
<00:00:02.680>Hi, <00:00:02.820>everyone. <00:00:02.960>I'm <00:00:03.100>Csenge, <00:00:03.240>an <00:00:03.380>AI <00:00:03.520>engineer, <00:00:03.660>and <00:00:03.800>this <00:00:03.940>talk <00:00:04.080>is

2
00:00:06.480 --> 00:00:10.120
<00:00:06.480>about <00:00:06.607>something <00:00:06.734>every <00:00:06.861>RAG <00:00:06.989>system <00:00:07.116>needs, <00:00:07.243>but <00:00:07.370>not <00:00:07.498>many <00:00:07.625>of <00:00:07.752>them <00:00:07.880>have,

3
00:00:10.620 --> 00:00:12.760
<00:00:10.620>that <00:00:10.899>is <00:00:11.180>a <00:00:11.459>proper <00:00:11.739>evaluation <00:00:12.020>pipeline.

4
00:00:13.780 --> 00:00:17.760
<00:00:13.780>A <00:00:13.920>few <00:00:14.059>words <00:00:14.200>about <00:00:14.340>myself. <00:00:14.479>I'm <00:00:14.620>working <00:00:14.760>at <00:00:14.899>Manfold <00:00:15.040>AI, <00:00:15.180>which

5
00:00:17.820 --> 00:00:19.740
<00:00:17.820>is <00:00:18.100>a <00:00:18.380>startup <00:00:18.660>based <00:00:18.940>in <00:00:19.220>Vienna,

6
00:00:20.920 --> 00:00:24.760
<00:00:20.920>and <00:00:21.060>our <00:00:21.200>main <00:00:21.340>objective <00:00:21.480>is <00:00:21.620>to <00:00:21.760>build <00:00:21.900>agentic <00:00:22.040>AI <00:00:22.180>systems <00:00:22.320>that

7
00:00:24.840 --> 00:00:28.100
<00:00:24.840>are <00:00:25.073>building <00:00:25.306>upon <00:00:25.540>RAG <00:00:25.773>for <00:00:26.006>enterprise <00:00:26.240>customers.

8
00:00:28.780 --> 00:00:32.729
<00:00:28.780>We <00:00:28.935>spend <00:00:29.091>a <00:00:29.246>lot <00:00:29.402>of <00:00:29.557>time <00:00:29.713>with <00:00:29.868>observability <00:00:30.024>and <00:00:30.180>evaluation,

9
00:00:32.820 --> 00:00:36.150
<00:00:32.820>and <00:00:32.975>I'm <00:00:33.131>very <00:00:33.286>happy <00:00:33.442>to <00:00:33.597>share <00:00:33.753>our <00:00:33.908>experience <00:00:34.064>on <00:00:34.220>this

10
00:00:36.160 --> 00:00:40.000
<00:00:36.160>platform. <00:00:36.276>By <00:00:36.393>the <00:00:36.510>end <00:00:36.626>of <00:00:36.743>this <00:00:36.860>talk, <00:00:36.976>I <00:00:37.093>hope <00:00:37.209>that <00:00:37.326>you'll <00:00:37.443>have <00:00:37.559>a

11
00:00:40.020 --> 00:00:43.960
<00:00:40.020>repeatable, <00:00:40.175>low-effort <00:00:40.331>way <00:00:40.486>to <00:00:40.642>measure <00:00:40.797>your <00:00:40.953>RAG <00:00:41.108>systems <00:00:41.264>using <00:00:41.420>your

12
00:00:44.020 --> 00:00:44.860
<00:00:44.020>own <00:00:44.775>documents

13
00:00:45.680 --> 00:00:47.559
<00:00:45.680>and <00:00:45.960>setting <00:00:46.240>up <00:00:46.519>your <00:00:46.800>own <00:00:47.080>pipelines.

14
00:00:48.280 --> 00:00:52.240
<00:00:48.280>In <00:00:48.367>this <00:00:48.455>talk, <00:00:48.542>I'm <00:00:48.630>going <00:00:48.717>to <00:00:48.805>lead <00:00:48.892>you <00:00:48.980>through <00:00:49.067>a <00:00:49.155>slide <00:00:49.242>deck, <00:00:49.330>but <00:00:49.417>I <00:00:49.505>will <00:00:49.592>also <00:00:49.680>show

15
00:00:52.380 --> 00:00:55.710
<00:00:52.380>you <00:00:52.555>a <00:00:52.730>notebook <00:00:52.905>with <00:00:53.080>code, <00:00:53.255>which <00:00:53.430>we'll <00:00:53.605>run <00:00:53.780>together.

16
00:00:55.780 --> 00:00:59.720
<00:00:55.780>And <00:00:55.887>of <00:00:55.995>course, <00:00:56.103>at <00:00:56.210>the <00:00:56.318>end, <00:00:56.426>you'll <00:00:56.533>have <00:00:56.641>access <00:00:56.749>to <00:00:56.856>my <00:00:56.964>GitHub <00:00:57.072>repository <00:00:57.180>so

17
00:00:59.760 --> 00:01:03.660
<00:00:59.760>that <00:00:59.876>you <00:00:59.993>can <00:01:00.110>rerun <00:01:00.226>this <00:01:00.343>notebook <00:01:00.460>or <00:01:00.576>refer <00:01:00.693>back <00:01:00.809>to <00:01:00.926>it <00:01:01.043>anytime <00:01:01.160>afterwards.

18
00:01:06.720 --> 00:01:09.800
<00:01:06.720>Let's <00:01:06.847>begin <00:01:06.974>with <00:01:07.101>something <00:01:07.229>that <00:01:07.356>many <00:01:07.483>of <00:01:07.610>you <00:01:07.738>might <00:01:07.865>be <00:01:07.992>familiar <00:01:08.120>with.

19
00:01:10.240 --> 00:01:13.980
<00:01:10.240>There <00:01:10.380>might <00:01:10.520>be <00:01:10.660>a <00:01:10.800>request <00:01:10.940>to <00:01:11.080>build <00:01:11.220>a <00:01:11.360>chatbot <00:01:11.500>that <00:01:11.640>is

20
00:01:14.020 --> 00:01:17.320
<00:01:14.020>based <00:01:14.195>on <00:01:14.369>a <00:01:14.545>knowledge <00:01:14.720>base. <00:01:14.895>And <00:01:15.070>so <00:01:15.244>you <00:01:15.420>begin

21
00:01:17.480 --> 00:01:21.250
<00:01:17.480>experimenting <00:01:17.635>and <00:01:17.791>you <00:01:17.946>set <00:01:18.102>up <00:01:18.257>an <00:01:18.413>LLM-based <00:01:18.568>chatbot <00:01:18.724>that <00:01:18.880>is

22
00:01:21.280 --> 00:01:24.480
<00:01:21.280>grounded <00:01:21.420>in <00:01:21.560>a <00:01:21.700>dataset. <00:01:21.840>You <00:01:21.980>try <00:01:22.120>it <00:01:22.260>out <00:01:22.400>and <00:01:22.540>everything <00:01:22.680>seems

23
00:01:24.540 --> 00:01:27.440
<00:01:24.539>fine, <00:01:24.695>so <00:01:24.851>you're <00:01:25.006>ready <00:01:25.162>to <00:01:25.317>roll <00:01:25.473>out <00:01:25.628>to <00:01:25.784>your <00:01:25.940>customers.

24
00:01:27.840 --> 00:01:31.580
<00:01:27.840>It <00:01:27.995>might <00:01:28.151>happen <00:01:28.306>that <00:01:28.462>sometime <00:01:28.617>later, <00:01:28.773>once <00:01:28.928>your <00:01:29.084>users <00:01:29.240>are

25
00:01:31.640 --> 00:01:35.360
<00:01:31.640>interacting <00:01:31.756>with <00:01:31.873>the <00:01:31.990>platform, <00:01:32.106>you <00:01:32.223>receive <00:01:32.340>the <00:01:32.456>news <00:01:32.573>that <00:01:32.690>the <00:01:32.806>system <00:01:32.923>might <00:01:33.040>be

26
00:01:35.400 --> 00:01:38.940
<00:01:35.400>hallucinating <00:01:35.516>here <00:01:35.633>and <00:01:35.750>there, <00:01:35.866>or <00:01:35.983>it <00:01:36.100>is <00:01:36.216>giving <00:01:36.333>you <00:01:36.450>answers <00:01:36.566>that <00:01:36.683>are <00:01:36.800>not

27
00:01:38.980 --> 00:01:40.740
<00:01:38.979>grounded <00:01:39.446>in <00:01:39.913>the <00:01:40.380>data.

28
00:01:42.020 --> 00:01:45.570
<00:01:42.020>So <00:01:42.147>you <00:01:42.274>notice <00:01:42.401>that <00:01:42.529>the <00:01:42.656>users <00:01:42.783>are <00:01:42.910>not <00:01:43.038>happy <00:01:43.165>with <00:01:43.292>your <00:01:43.420>system,

29
00:01:47.460 --> 00:01:50.760
<00:01:47.460>and <00:01:47.587>therefore, <00:01:47.714>I <00:01:47.841>would <00:01:47.969>like <00:01:48.096>to <00:01:48.223>introduce <00:01:48.350>you <00:01:48.478>to <00:01:48.605>a <00:01:48.732>way <00:01:48.860>of

30
00:01:51.160 --> 00:01:54.720
<00:01:51.160>measuring <00:01:51.300>your <00:01:51.440>RAG <00:01:51.580>systems <00:01:51.720>even <00:01:51.860>before <00:01:52.000>they <00:01:52.140>are <00:01:52.280>pushed <00:01:52.420>into <00:01:52.560>production

31
00:01:55.340 --> 00:01:59.280
<00:01:55.340>and <00:01:55.495>catch <00:01:55.651>the <00:01:55.806>issues <00:01:55.962>and <00:01:56.117>errors <00:01:56.273>before <00:01:56.428>your <00:01:56.584>users <00:01:56.740>do.

32
00:02:02.660 --> 00:02:06.480
<00:02:02.660>A <00:02:02.776>quick <00:02:02.893>refresher <00:02:03.009>on <00:02:03.126>RAG. <00:02:03.243>Let's <00:02:03.360>have <00:02:03.476>a <00:02:03.593>look <00:02:03.710>at <00:02:03.826>how <00:02:03.943>a <00:02:04.060>RAG

33
00:02:06.540 --> 00:02:10.301
<00:02:06.540>pipeline <00:02:06.740>actually <00:02:06.940>works. <00:02:07.140>So <00:02:07.340>Retrieval <00:02:07.540>Augmented <00:02:07.740>Generation <00:02:07.940>is

34
00:02:10.361 --> 00:02:12.020
<00:02:10.360>composed <00:02:10.710>of <00:02:11.060>a <00:02:11.411>few <00:02:11.761>steps.

35
00:02:12.840 --> 00:02:16.280
<00:02:12.840>We <00:02:12.995>tend <00:02:13.151>to <00:02:13.306>begin <00:02:13.462>with <00:02:13.617>documents, <00:02:13.773>which <00:02:13.928>could <00:02:14.084>be <00:02:14.240>PDFs,

36
00:02:16.780 --> 00:02:20.140
<00:02:16.780>DOCX <00:02:17.013>files, <00:02:17.246>Markdown <00:02:17.480>files, <00:02:17.713>and <00:02:17.946>so <00:02:18.180>on.

37
00:02:20.200 --> 00:02:23.000
<00:02:20.200>Then <00:02:20.433>we <00:02:20.666>split <00:02:20.899>these <00:02:21.133>documents <00:02:21.366>into <00:02:21.600>chunks.

38
00:02:23.680 --> 00:02:27.190
<00:02:23.680>We <00:02:23.820>want <00:02:23.960>to <00:02:24.100>convert <00:02:24.240>them <00:02:24.380>into <00:02:24.520>some <00:02:24.660>kind <00:02:24.800>of <00:02:24.940>a <00:02:25.080>numerical

39
00:02:27.220 --> 00:02:30.300
<00:02:27.220>representation <00:02:27.375>so <00:02:27.531>that <00:02:27.686>large <00:02:27.842>language <00:02:27.997>models <00:02:28.153>can <00:02:28.308>work <00:02:28.464>with <00:02:28.620>it.

40
00:02:30.980 --> 00:02:33.560
<00:02:30.980>That's <00:02:31.213>why <00:02:31.446>we <00:02:31.679>convert <00:02:31.913>them <00:02:32.146>into <00:02:32.380>vectors.

41
00:02:34.160 --> 00:02:38.100
<00:02:34.160>And <00:02:34.299>then <00:02:34.440>these <00:02:34.579>embeddings <00:02:34.720>are <00:02:34.859>stored <00:02:35.000>in <00:02:35.140>the <00:02:35.280>vector <00:02:35.420>store <00:02:35.560>so

42
00:02:38.160 --> 00:02:40.829
<00:02:38.160>that <00:02:38.440>the <00:02:38.720>LLMs <00:02:39.000>can <00:02:39.280>index <00:02:39.560>and

43
00:02:41.780 --> 00:02:42.320
<00:02:41.780>search <00:02:42.266>them.

44
00:02:43.880 --> 00:02:47.140
<00:02:43.880>Once <00:02:44.055>you <00:02:44.230>have <00:02:44.405>all <00:02:44.579>of <00:02:44.755>the <00:02:44.930>preparation <00:02:45.105>done <00:02:45.280>once,

45
00:02:47.820 --> 00:02:50.320
<00:02:47.820>then <00:02:48.019>you <00:02:48.220>are <00:02:48.420>ready <00:02:48.620>to <00:02:48.820>use <00:02:49.019>the <00:02:49.220>system.

46
00:02:50.530 --> 00:02:54.280
<00:02:50.530>And <00:02:50.657>what <00:02:50.784>happens <00:02:50.911>is <00:02:51.039>that <00:02:51.166>the <00:02:51.293>user <00:02:51.420>sends <00:02:51.548>a <00:02:51.675>question <00:02:51.802>to <00:02:51.930>this

47
00:02:54.340 --> 00:02:58.240
<00:02:54.340>system. <00:02:54.495>The <00:02:54.651>question <00:02:54.806>gets <00:02:54.962>embedded <00:02:55.117>the <00:02:55.273>same <00:02:55.428>way <00:02:55.584>as <00:02:55.740>the

48
00:02:58.260 --> 00:03:01.600
<00:02:58.260>source <00:02:58.459>documents <00:02:58.660>were. <00:02:58.859>Then <00:02:59.060>comes <00:02:59.260>the <00:02:59.459>retrieval <00:02:59.660>step.

49
00:03:02.240 --> 00:03:05.160
<00:03:02.240>In <00:03:02.395>the <00:03:02.551>retrieval <00:03:02.706>step, <00:03:02.862>the <00:03:03.017>LLM <00:03:03.173>fetches <00:03:03.328>the <00:03:03.484>right <00:03:03.640>context,

50
00:03:05.560 --> 00:03:08.780
<00:03:05.560>meaning <00:03:05.760>it <00:03:05.960>tries <00:03:06.160>to <00:03:06.360>find <00:03:06.560>the <00:03:06.760>most <00:03:06.960>relevant

51
00:03:09.240 --> 00:03:12.980
<00:03:09.240>passages <00:03:09.380>from <00:03:09.520>the <00:03:09.660>text <00:03:09.800>based <00:03:09.940>on <00:03:10.080>some <00:03:10.220>kind <00:03:10.360>of <00:03:10.500>ranking <00:03:10.640>mechanism.

52
00:03:14.080 --> 00:03:17.820
<00:03:14.080>And <00:03:14.235>once <00:03:14.391>the <00:03:14.546>top <00:03:14.702>chunks <00:03:14.857>were <00:03:15.013>retrieved, <00:03:15.168>then <00:03:15.324>it <00:03:15.480>is

53
00:03:17.860 --> 00:03:18.960
<00:03:17.860>added <00:03:18.190>to <00:03:18.520>a <00:03:18.850>prompt

54
00:03:19.880 --> 00:03:22.520
<00:03:19.880>where <00:03:20.113>the <00:03:20.346>generation <00:03:20.579>step <00:03:20.813>comes <00:03:21.046>into <00:03:21.280>play.

55
00:03:22.820 --> 00:03:26.400
<00:03:22.820>The <00:03:22.947>LLM <00:03:23.074>is <00:03:23.201>going <00:03:23.329>to <00:03:23.456>compose <00:03:23.583>an <00:03:23.710>answer <00:03:23.838>based <00:03:23.965>on <00:03:24.092>the <00:03:24.220>user's

56
00:03:26.460 --> 00:03:27.450
<00:03:26.460>original <00:03:27.351>question,

57
00:03:28.420 --> 00:03:30.680
<00:03:28.420>the <00:03:29.120>relevant <00:03:29.820>chunks,

58
00:03:31.820 --> 00:03:35.620
<00:03:31.820>and <00:03:31.959>its <00:03:32.100>own <00:03:32.239>prompt. <00:03:32.380>And <00:03:32.519>then <00:03:32.660>after <00:03:32.799>that, <00:03:32.940>the <00:03:33.079>answer <00:03:33.220>is

59
00:03:35.680 --> 00:03:36.940
<00:03:35.680>sent <00:03:35.963>back <00:03:36.247>to <00:03:36.530>the <00:03:36.814>user.

60
00:03:38.000 --> 00:03:41.640
<00:03:38.000>So <00:03:38.155>what <00:03:38.311>is <00:03:38.466>measurable <00:03:38.622>in <00:03:38.777>this <00:03:38.933>case <00:03:39.088>is <00:03:39.244>the <00:03:39.400>retrieval,

61
00:03:42.220 --> 00:03:45.760
<00:03:42.220>referring <00:03:42.375>to <00:03:42.531>whether <00:03:42.686>we <00:03:42.842>managed <00:03:42.997>to <00:03:43.153>catch <00:03:43.308>all <00:03:43.464>the <00:03:43.620>relevant

62
00:03:46.040 --> 00:03:49.380
<00:03:46.040>chunks, <00:03:46.179>and <00:03:46.320>also <00:03:46.459>the <00:03:46.600>generation, <00:03:46.739>if <00:03:46.880>the <00:03:47.019>LLM <00:03:47.160>did <00:03:47.299>a <00:03:47.440>good

63
00:03:49.440 --> 00:03:53.320
<00:03:49.440>job <00:03:49.567>at <00:03:49.694>putting <00:03:49.821>the <00:03:49.949>context <00:03:50.076>together <00:03:50.203>and <00:03:50.330>then <00:03:50.458>refining <00:03:50.585>it <00:03:50.712>to <00:03:50.840>a

64
00:03:53.360 --> 00:03:54.840
<00:03:53.360>final <00:03:53.693>answer <00:03:54.026>to <00:03:54.359>the <00:03:54.692>user.

65
00:03:56.540 --> 00:04:00.180
<00:03:56.540>So <00:03:56.773>RAG <00:03:57.006>can <00:03:57.239>fail <00:03:57.473>independently. <00:03:57.706>That's <00:03:57.940>why

66
00:04:00.200 --> 00:04:03.600
<00:04:00.200>evaluation <00:04:00.339>is <00:04:00.480>tricky, <00:04:00.619>because <00:04:00.760>there <00:04:00.899>is <00:04:01.040>many <00:04:01.179>components <00:04:01.320>that <00:04:01.459>could <00:04:01.600>be

67
00:04:03.640 --> 00:04:07.200
<00:04:03.640>measured. <00:04:03.815>There <00:04:03.989>are <00:04:04.165>two <00:04:04.339>main <00:04:04.515>failure <00:04:04.690>surfaces <00:04:04.864>for <00:04:05.040>RAG,

68
00:04:07.549 --> 00:04:10.090
<00:04:07.549>which <00:04:07.782>are <00:04:08.015>the <00:04:08.249>retrieval <00:04:08.482>and <00:04:08.715>the <00:04:08.949>generation.

69
00:04:11.700 --> 00:04:15.580
<00:04:11.700>It <00:04:11.816>might <00:04:11.933>be <00:04:12.049>that <00:04:12.166>a <00:04:12.283>good <00:04:12.399>answer <00:04:12.516>comes <00:04:12.633>from <00:04:12.750>an <00:04:12.866>incorrect <00:04:12.983>retrieval <00:04:13.100>mechanism,

70
00:04:15.600 --> 00:04:19.351
<00:04:15.600>and <00:04:15.755>vice <00:04:15.911>versa. <00:04:16.066>Therefore, <00:04:16.222>it's <00:04:16.377>important <00:04:16.533>to <00:04:16.688>measure <00:04:16.844>these <00:04:17.000>steps

71
00:04:20.079 --> 00:04:20.661
separately.

72
00:04:21.860 --> 00:04:25.480
<00:04:21.860>It <00:04:22.000>can <00:04:22.140>happen <00:04:22.280>that <00:04:22.420>there <00:04:22.560>is <00:04:22.700>a <00:04:22.840>good <00:04:22.980>retrieval <00:04:23.120>mechanism, <00:04:23.260>but

73
00:04:25.500 --> 00:04:27.660
<00:04:25.500>something <00:04:25.733>goes <00:04:25.966>wrong <00:04:26.200>with <00:04:26.433>the <00:04:26.666>generation <00:04:26.900>step.

74
00:04:27.750 --> 00:04:31.300
<00:04:27.750>This <00:04:27.925>is <00:04:28.100>usually <00:04:28.275>the <00:04:28.450>model's <00:04:28.625>fault, <00:04:28.800>meaning <00:04:28.975>the <00:04:29.150>context

75
00:04:31.440 --> 00:04:34.520
<00:04:31.440>retrieval <00:04:31.615>chunks <00:04:31.790>were <00:04:31.965>correct, <00:04:32.140>but <00:04:32.315>the <00:04:32.490>generated <00:04:32.665>answer <00:04:32.840>not.

76
00:04:34.560 --> 00:04:38.510
<00:04:34.560>So <00:04:34.700>in <00:04:34.840>this <00:04:34.980>case, <00:04:35.120>it <00:04:35.260>could <00:04:35.400>be <00:04:35.540>something <00:04:35.680>to <00:04:35.820>do <00:04:35.960>with

77
00:04:38.560 --> 00:04:39.520
<00:04:38.560>the <00:04:38.992>prompt <00:04:39.424>itself.

78
00:04:40.580 --> 00:04:44.480
<00:04:40.580>In <00:04:40.696>another <00:04:40.813>scenario, <00:04:40.930>it <00:04:41.046>can <00:04:41.163>be <00:04:41.280>that <00:04:41.396>the <00:04:41.513>generation <00:04:41.630>was <00:04:41.746>correct, <00:04:41.863>however, <00:04:41.979>the

79
00:04:44.500 --> 00:04:47.080
<00:04:44.500>retrieval, <00:04:44.700>the <00:04:44.900>chunks <00:04:45.100>were <00:04:45.300>not <00:04:45.500>the <00:04:45.700>right <00:04:45.900>ones.

80
00:04:47.870 --> 00:04:51.300
<00:04:47.870>This <00:04:48.010>is <00:04:48.150>very <00:04:48.290>dangerous <00:04:48.430>because <00:04:48.570>it <00:04:48.710>means <00:04:48.850>that <00:04:48.990>it <00:04:49.130>can <00:04:49.270>break

81
00:04:51.400 --> 00:04:54.290
<00:04:51.400>silently <00:04:51.750>without <00:04:52.099>us <00:04:52.450>noticing <00:04:52.799>anything.

82
00:04:54.860 --> 00:04:58.140
<00:04:54.860>And <00:04:54.976>of <00:04:55.093>course, <00:04:55.210>it <00:04:55.326>might <00:04:55.443>be <00:04:55.560>that <00:04:55.676>both <00:04:55.793>retrieval <00:04:55.910>and <00:04:56.026>generation <00:04:56.143>go <00:04:56.260>wrong.

83
00:04:58.640 --> 00:05:02.200
<00:04:58.640>Then <00:04:58.767>we <00:04:58.894>need <00:04:59.021>to <00:04:59.149>go <00:04:59.276>step <00:04:59.403>by <00:04:59.530>step <00:04:59.658>and <00:04:59.785>correct <00:04:59.912>the <00:05:00.039>system.

84
00:05:04.440 --> 00:05:08.220
<00:05:04.440>So <00:05:04.595>why <00:05:04.751>is <00:05:04.906>it <00:05:05.062>so <00:05:05.217>difficult <00:05:05.373>to <00:05:05.528>measure <00:05:05.684>these <00:05:05.840>systems,

85
00:05:08.440 --> 00:05:11.940
<00:05:08.440>and <00:05:08.673>why <00:05:08.906>is <00:05:09.140>it <00:05:09.373>necessary <00:05:09.606>to <00:05:09.840>have

86
00:05:12.160 --> 00:05:12.440
this

87
00:05:13.240 --> 00:05:17.060
<00:05:13.240>done? <00:05:13.473>Well, <00:05:13.706>evaluation <00:05:13.940>is <00:05:14.173>difficult <00:05:14.406>because <00:05:14.640>for

88
00:05:17.260 --> 00:05:20.760
<00:05:17.260>a <00:05:17.387>good <00:05:17.514>evaluation, <00:05:17.641>you <00:05:17.769>of <00:05:17.896>course <00:05:18.023>need <00:05:18.150>a <00:05:18.278>dataset, <00:05:18.405>a <00:05:18.532>test <00:05:18.659>set,

89
00:05:21.620 --> 00:05:25.100
<00:05:21.620>and <00:05:21.775>for <00:05:21.931>good <00:05:22.086>testing, <00:05:22.242>we <00:05:22.397>rely <00:05:22.553>on <00:05:22.708>volume <00:05:22.864>and <00:05:23.020>also

90
00:05:25.200 --> 00:05:25.700
coverage.

91
00:05:26.560 --> 00:05:30.540
<00:05:26.560>What <00:05:26.687>I <00:05:26.814>mean <00:05:26.941>here <00:05:27.069>is <00:05:27.196>that <00:05:27.323>if <00:05:27.450>an <00:05:27.578>engineer <00:05:27.705>puts <00:05:27.832>together <00:05:27.960>five

92
00:05:30.640 --> 00:05:34.400
<00:05:30.640>simple <00:05:30.795>questions <00:05:30.951>and <00:05:31.106>starts <00:05:31.262>shooting <00:05:31.417>the <00:05:31.573>RAG <00:05:31.728>system, <00:05:31.884>that <00:05:32.039>doesn't

93
00:05:34.480 --> 00:05:38.380
<00:05:34.480>mean <00:05:34.620>proper <00:05:34.760>evaluation <00:05:34.900>because <00:05:35.040>we <00:05:35.180>cannot <00:05:35.320>have <00:05:35.460>the <00:05:35.600>proper <00:05:35.740>coverage, <00:05:35.880>we

94
00:05:38.420 --> 00:05:42.280
<00:05:38.420>cannot <00:05:38.547>prepare <00:05:38.674>for <00:05:38.801>all <00:05:38.929>the <00:05:39.056>variety <00:05:39.183>of <00:05:39.310>questions <00:05:39.438>the <00:05:39.565>users <00:05:39.692>might <00:05:39.820>ask.

95
00:05:43.440 --> 00:05:47.340
<00:05:43.440>Also, <00:05:43.615>once <00:05:43.790>you <00:05:43.965>start <00:05:44.140>evaluating <00:05:44.315>and <00:05:44.490>working <00:05:44.665>on <00:05:44.840>your

96
00:05:47.840 --> 00:05:51.180
<00:05:47.840>RAG <00:05:48.015>setup, <00:05:48.190>you <00:05:48.365>might <00:05:48.540>want <00:05:48.715>to <00:05:48.890>change <00:05:49.065>some <00:05:49.240>things.

97
00:05:51.720 --> 00:05:53.220
<00:05:51.720>You <00:05:51.945>might <00:05:52.170>want <00:05:52.395>to <00:05:52.620>tweak <00:05:52.845>the <00:05:53.070>prompt

98
00:05:54.180 --> 00:05:58.000
<00:05:54.180>a <00:05:54.287>little <00:05:54.395>bit <00:05:54.503>and <00:05:54.610>then <00:05:54.718>see <00:05:54.826>whether <00:05:54.933>that <00:05:55.041>is <00:05:55.149>improving <00:05:55.256>or <00:05:55.364>degrading <00:05:55.472>your <00:05:55.580>system.

99
00:05:58.620 --> 00:06:01.300
<00:05:58.620>Here <00:05:58.853>again, <00:05:59.086>evaluation <00:05:59.320>is <00:05:59.553>an <00:05:59.786>important <00:06:00.020>step.

100
00:06:01.920 --> 00:06:05.880
<00:06:01.920>Also, <00:06:02.027>it's <00:06:02.135>a <00:06:02.243>good <00:06:02.350>idea <00:06:02.458>to <00:06:02.566>start <00:06:02.673>evaluation <00:06:02.781>early <00:06:02.889>so <00:06:02.996>that <00:06:03.104>you <00:06:03.212>can <00:06:03.320>have

101
00:06:05.920 --> 00:06:09.840
<00:06:05.920>a <00:06:06.095>baseline <00:06:06.270>of <00:06:06.445>system <00:06:06.620>performance, <00:06:06.795>and <00:06:06.970>these <00:06:07.145>scores <00:06:07.320>can

102
00:06:09.900 --> 00:06:13.800
<00:06:09.900>help <00:06:10.055>you <00:06:10.211>along <00:06:10.366>the <00:06:10.522>way <00:06:10.677>to <00:06:10.833>compare <00:06:10.988>whether <00:06:11.144>system <00:06:11.299>A

103
00:06:13.820 --> 00:06:14.980
<00:06:13.820>or <00:06:14.168>B <00:06:14.516>works <00:06:14.864>better.

104
00:06:16.944 --> 00:06:20.584
<00:06:16.944>So <00:06:17.084>as <00:06:17.224>mentioned, <00:06:17.364>we <00:06:17.504>need <00:06:17.644>a <00:06:17.784>test <00:06:17.924>set <00:06:18.064>system <00:06:18.204>to <00:06:18.344>score.

105
00:06:21.364 --> 00:06:24.243
<00:06:21.364>And <00:06:21.480>let's <00:06:21.597>have <00:06:21.714>a <00:06:21.830>look <00:06:21.947>at <00:06:22.063>what <00:06:22.180>is <00:06:22.297>actually <00:06:22.414>in <00:06:22.530>a <00:06:22.647>test <00:06:22.763>set.

106
00:06:24.284 --> 00:06:26.844
<00:06:24.284>Of <00:06:24.459>course, <00:06:24.634>we <00:06:24.808>would <00:06:24.984>like <00:06:25.159>to <00:06:25.334>have <00:06:25.509>questions <00:06:25.683>defined,

107
00:06:27.804 --> 00:06:31.484
<00:06:27.804>variety <00:06:27.931>of <00:06:28.058>complexity <00:06:28.185>there. <00:06:28.313>We <00:06:28.440>would <00:06:28.567>like <00:06:28.694>to <00:06:28.822>see <00:06:28.949>what <00:06:29.076>are <00:06:29.203>the

108
00:06:31.524 --> 00:06:35.504
<00:06:31.524>relevant <00:06:31.679>chunks <00:06:31.835>that <00:06:31.990>will <00:06:32.146>serve <00:06:32.301>the <00:06:32.457>model <00:06:32.612>to <00:06:32.768>answer <00:06:32.924>that

109
00:06:35.584 --> 00:06:36.064
question.

110
00:06:37.184 --> 00:06:40.664
<00:06:37.183>And <00:06:37.311>of <00:06:37.438>course, <00:06:37.565>we <00:06:37.693>need <00:06:37.820>gold <00:06:37.947>labels <00:06:38.074>or <00:06:38.202>some <00:06:38.329>kind <00:06:38.456>of <00:06:38.583>output,

111
00:06:41.324 --> 00:06:43.464
<00:06:41.324>what <00:06:41.557>we <00:06:41.790>expect <00:06:42.024>the <00:06:42.257>model <00:06:42.490>to <00:06:42.724>return.

112
00:06:44.564 --> 00:06:48.484
<00:06:44.564>When <00:06:44.704>you <00:06:44.844>want <00:06:44.984>to <00:06:45.124>handcraft <00:06:45.264>a <00:06:45.404>test <00:06:45.544>data <00:06:45.684>set <00:06:45.824>like <00:06:45.964>that,

113
00:06:48.564 --> 00:06:51.884
<00:06:48.564>you <00:06:48.680>might <00:06:48.797>run <00:06:48.914>into <00:06:49.030>some <00:06:49.147>issues <00:06:49.264>because <00:06:49.380>it <00:06:49.497>can <00:06:49.614>be <00:06:49.730>very <00:06:49.847>slow <00:06:49.964>and

114
00:06:51.944 --> 00:06:55.524
<00:06:51.944>expensive <00:06:52.099>to <00:06:52.255>do <00:06:52.410>this <00:06:52.566>by <00:06:52.721>hand. <00:06:52.877>You'll <00:06:53.032>need <00:06:53.188>to <00:06:53.344>hire

115
00:06:55.644 --> 00:06:59.144
<00:06:55.644>subject <00:06:55.844>matter <00:06:56.044>experts, <00:06:56.244>especially <00:06:56.444>if <00:06:56.644>your <00:06:56.844>domain <00:06:57.044>is

116
00:06:59.204 --> 00:07:02.984
<00:06:59.204>complex, <00:06:59.379>and <00:06:59.554>of <00:06:59.729>course, <00:06:59.904>if <00:07:00.079>humans <00:07:00.254>do <00:07:00.429>it, <00:07:00.604>it

117
00:07:03.024 --> 00:07:06.204
<00:07:03.024>might <00:07:03.304>be <00:07:03.584>biased <00:07:03.864>towards <00:07:04.144>particular <00:07:04.424>questions.

118
00:07:06.604 --> 00:07:10.024
<00:07:06.604>So <00:07:06.731>it's <00:07:06.858>really <00:07:06.985>difficult <00:07:07.113>to <00:07:07.240>come <00:07:07.367>up <00:07:07.494>with <00:07:07.622>good <00:07:07.749>volume <00:07:07.876>and <00:07:08.003>good

119
00:07:10.124 --> 00:07:10.664
coverage.

120
00:07:11.864 --> 00:07:15.524
<00:07:11.864>So <00:07:11.964>this <00:07:12.063>might <00:07:12.164>leave <00:07:12.263>you <00:07:12.364>to <00:07:12.464>wonder, <00:07:12.563>is <00:07:12.664>there <00:07:12.763>a <00:07:12.864>way <00:07:12.964>to <00:07:13.063>handle <00:07:13.164>volume <00:07:13.263>and

121
00:07:15.604 --> 00:07:19.244
<00:07:15.604>coverage <00:07:15.804>at <00:07:16.003>the <00:07:16.204>same <00:07:16.404>time <00:07:16.604>without <00:07:16.804>humans <00:07:17.003>being

122
00:07:19.304 --> 00:07:21.334
<00:07:19.304>involved <00:07:19.537>in <00:07:19.770>this <00:07:20.003>process <00:07:20.237>at <00:07:20.470>the <00:07:20.703>beginning?

123
00:07:23.084 --> 00:07:26.384
<00:07:23.084>And <00:07:23.211>the <00:07:23.338>answer <00:07:23.465>is <00:07:23.593>there <00:07:23.720>is <00:07:23.847>a <00:07:23.974>way. <00:07:24.102>This <00:07:24.229>is <00:07:24.356>where <00:07:24.484>synthetic

124
00:07:26.464 --> 00:07:30.224
<00:07:26.464>generation, <00:07:26.697>test <00:07:26.930>data <00:07:27.164>generation <00:07:27.397>comes <00:07:27.630>into <00:07:27.864>the

125
00:07:30.304 --> 00:07:30.664
play.

126
00:07:32.664 --> 00:07:36.584
<00:07:32.664>The <00:07:32.771>knowledge <00:07:32.879>already <00:07:32.987>lives <00:07:33.094>in <00:07:33.202>your <00:07:33.310>documents, <00:07:33.417>so <00:07:33.525>the <00:07:33.633>questions <00:07:33.740>and <00:07:33.848>the <00:07:33.956>answers <00:07:34.063>can

127
00:07:36.624 --> 00:07:40.334
<00:07:36.624>be <00:07:36.724>generated <00:07:36.824>from <00:07:36.924>them. <00:07:37.024>But <00:07:37.124>there <00:07:37.224>is <00:07:37.324>a <00:07:37.424>right <00:07:37.524>and <00:07:37.624>a <00:07:37.724>wrong <00:07:37.824>way <00:07:37.924>of <00:07:38.024>doing

128
00:07:40.364 --> 00:07:44.324
<00:07:40.364>this. <00:07:40.539>What <00:07:40.714>you <00:07:40.888>might <00:07:41.063>intuitively <00:07:41.239>think <00:07:41.414>of <00:07:41.589>is <00:07:41.763>to

129
00:07:44.384 --> 00:07:47.944
<00:07:44.384>go <00:07:44.524>ahead <00:07:44.664>with <00:07:44.804>a <00:07:44.944>shortcut, <00:07:45.084>choose <00:07:45.224>an <00:07:45.364>LLM <00:07:45.504>of <00:07:45.644>your <00:07:45.784>choice,

130
00:07:48.484 --> 00:07:51.844
<00:07:48.484>just <00:07:48.659>let <00:07:48.834>the <00:07:49.008>LLM <00:07:49.183>iterate <00:07:49.359>over <00:07:49.534>each <00:07:49.709>chunk <00:07:49.883>of

131
00:07:51.904 --> 00:07:55.344
<00:07:51.904>your <00:07:52.079>knowledge <00:07:52.254>base, <00:07:52.429>and <00:07:52.604>then <00:07:52.779>generate <00:07:52.954>questions <00:07:53.129>from <00:07:53.304>them.

132
00:07:56.504 --> 00:07:59.884
<00:07:56.504>This <00:07:56.679>is <00:07:56.854>naive, <00:07:57.029>because <00:07:57.204>there <00:07:57.379>is <00:07:57.554>a <00:07:57.729>chance <00:07:57.904>that

133
00:08:00.324 --> 00:08:02.164
<00:08:00.324>there's <00:08:00.674>not <00:08:01.024>deep <00:08:01.374>enough <00:08:01.724>complexity.

134
00:08:02.744 --> 00:08:06.584
<00:08:02.744>You're <00:08:02.860>simply <00:08:02.977>asking <00:08:03.094>one <00:08:03.210>question <00:08:03.327>per <00:08:03.444>chunk, <00:08:03.560>so <00:08:03.677>you <00:08:03.794>do <00:08:03.910>not <00:08:04.027>have <00:08:04.144>questions

135
00:08:06.664 --> 00:08:06.884
that

136
00:08:08.104 --> 00:08:10.924
<00:08:08.104>require <00:08:08.244>you <00:08:08.383>to <00:08:08.524>look <00:08:08.664>at <00:08:08.804>multiple <00:08:08.943>chunks <00:08:09.084>at <00:08:09.224>the <00:08:09.364>same <00:08:09.503>time.

137
00:08:11.404 --> 00:08:13.884
<00:08:11.404>The <00:08:11.604>questions <00:08:11.804>tend <00:08:12.004>to <00:08:12.204>be <00:08:12.404>repetitive <00:08:12.604>and <00:08:12.804>shallow,

138
00:08:14.864 --> 00:08:18.044
<00:08:14.864>and <00:08:14.971>they <00:08:15.079>are <00:08:15.187>lacking <00:08:15.294>the <00:08:15.402>variety <00:08:15.510>that <00:08:15.617>we <00:08:15.725>would <00:08:15.833>expect <00:08:15.940>from <00:08:16.048>a <00:08:16.156>test <00:08:16.263>set.

139
00:08:18.704 --> 00:08:21.984
<00:08:18.704>Therefore, <00:08:18.904>using <00:08:19.104>this <00:08:19.304>method <00:08:19.504>is <00:08:19.704>great <00:08:19.904>for <00:08:20.104>volume.

140
00:08:22.044 --> 00:08:25.484
<00:08:22.044>You <00:08:22.183>could <00:08:22.323>generate <00:08:22.464>hundreds <00:08:22.604>of <00:08:22.744>questions, <00:08:22.883>but <00:08:23.024>the <00:08:23.164>coverage <00:08:23.304>is <00:08:23.443>not

141
00:08:25.523 --> 00:08:29.004
<00:08:25.523>appropriate. <00:08:25.698>This <00:08:25.873>is <00:08:26.048>where <00:08:26.223>RAGAS, <00:08:26.398>which <00:08:26.573>is <00:08:26.748>an <00:08:26.923>open-source

142
00:08:29.064 --> 00:08:32.204
<00:08:29.064>framework <00:08:29.204>that <00:08:29.344>I'll <00:08:29.484>use <00:08:29.624>in <00:08:29.764>this <00:08:29.904>talk, <00:08:30.044>comes <00:08:30.184>into <00:08:30.324>the <00:08:30.464>play.

143
00:08:32.985 --> 00:08:36.725
<00:08:32.985>It <00:08:33.092>allows <00:08:33.200>you <00:08:33.308>to <00:08:33.415>build <00:08:33.523>a <00:08:33.631>knowledge <00:08:33.738>graph <00:08:33.846>from <00:08:33.954>your <00:08:34.061>documents <00:08:34.169>and <00:08:34.277>then <00:08:34.385>generate

144
00:08:36.784 --> 00:08:40.154
<00:08:36.784>questions <00:08:36.958>using <00:08:37.134>that <00:08:37.308>knowledge <00:08:37.484>graph <00:08:37.659>that <00:08:37.833>you <00:08:38.009>have <00:08:38.183>prepared.

145
00:08:40.824 --> 00:08:44.164
<00:08:40.823>The <00:08:40.963>reason <00:08:41.103>why <00:08:41.243>I <00:08:41.383>recommend <00:08:41.524>this <00:08:41.664>is <00:08:41.804>because <00:08:41.943>you <00:08:42.083>can <00:08:42.223>generate

146
00:08:44.744 --> 00:08:48.504
<00:08:44.744>not <00:08:44.871>only <00:08:44.998>simple <00:08:45.125>but <00:08:45.253>complex <00:08:45.380>questions <00:08:45.507>where <00:08:45.634>you <00:08:45.762>need <00:08:45.889>to <00:08:46.016>look <00:08:46.144>at

147
00:08:48.544 --> 00:08:51.104
<00:08:48.544>multiple <00:08:48.699>chunks <00:08:48.855>at <00:08:49.010>the <00:08:49.166>same <00:08:49.321>time <00:08:49.477>to <00:08:49.632>answer <00:08:49.788>the <00:08:49.944>query.

148
00:08:51.864 --> 00:08:55.104
<00:08:51.864>You <00:08:52.019>can <00:08:52.175>set <00:08:52.330>up <00:08:52.486>specific <00:08:52.641>and <00:08:52.797>abstract <00:08:52.952>questions <00:08:53.108>as <00:08:53.264>well.

149
00:08:55.124 --> 00:08:58.964
<00:08:55.124>Abstract <00:08:55.299>meaning <00:08:55.474>that <00:08:55.649>there <00:08:55.824>is <00:08:55.999>not <00:08:56.174>a <00:08:56.349>one-to-one <00:08:56.524>exact

150
00:08:59.044 --> 00:09:01.354
<00:08:59.044>sentence <00:08:59.244>in <00:08:59.444>the <00:08:59.644>knowledge <00:08:59.843>base <00:09:00.044>that <00:09:00.244>answers <00:09:00.444>that

151
00:09:01.384 --> 00:09:05.104
<00:09:01.384>question. <00:09:01.734>You <00:09:02.084>will <00:09:02.434>also <00:09:02.784>see

152
00:09:05.114 --> 00:09:06.584
<00:09:05.114>exactly <00:09:05.444>what <00:09:05.775>were <00:09:06.106>the <00:09:06.436>chunks

153
00:09:07.504 --> 00:09:11.104
<00:09:07.504>relevant <00:09:07.644>to <00:09:07.784>answer <00:09:07.924>that <00:09:08.064>question, <00:09:08.204>and <00:09:08.344>you <00:09:08.484>can <00:09:08.624>also <00:09:08.764>define <00:09:08.904>the

154
00:09:11.144 --> 00:09:12.464
persona

155
00:09:13.604 --> 00:09:17.484
<00:09:13.604>over <00:09:13.704>time, <00:09:13.804>meaning <00:09:13.904>you <00:09:14.004>can <00:09:14.104>define <00:09:14.204>who <00:09:14.304>would <00:09:14.404>be <00:09:14.504>the <00:09:14.604>users <00:09:14.704>of <00:09:14.804>your <00:09:14.904>RAG <00:09:15.004>system,

156
00:09:18.144 --> 00:09:21.674
<00:09:18.144>and <00:09:18.299>this <00:09:18.455>setup <00:09:18.610>allows <00:09:18.766>you <00:09:18.921>to <00:09:19.077>cover <00:09:19.232>both <00:09:19.388>volume <00:09:19.544>and

157
00:09:21.764 --> 00:09:22.304
coverage.

158
00:09:23.224 --> 00:09:27.084
<00:09:23.224>So <00:09:23.399>why <00:09:23.574>RAGAS <00:09:23.749>exactly? <00:09:23.924>What's <00:09:24.099>happening <00:09:24.274>in <00:09:24.449>this <00:09:24.624>framework?

159
00:09:27.984 --> 00:09:31.644
<00:09:27.984>As <00:09:28.139>mentioned, <00:09:28.295>it's <00:09:28.450>open <00:09:28.606>source <00:09:28.761>available <00:09:28.917>on <00:09:29.072>GitHub, <00:09:29.228>and <00:09:29.384>it

160
00:09:32.004 --> 00:09:35.224
<00:09:32.004>can <00:09:32.179>be <00:09:32.354>used <00:09:32.529>for <00:09:32.704>evaluating <00:09:32.879>RAG <00:09:33.054>and <00:09:33.229>agentic <00:09:33.404>applications.

161
00:09:35.464 --> 00:09:38.974
<00:09:35.464>It <00:09:35.638>very <00:09:35.814>nicely <00:09:35.988>integrates <00:09:36.164>with <00:09:36.339>LangChain <00:09:36.513>and <00:09:36.689>LangGraph <00:09:36.863>and

162
00:09:39.064 --> 00:09:42.824
<00:09:39.064>LlamaIndex, <00:09:39.238>and <00:09:39.414>also <00:09:39.589>it <00:09:39.764>can <00:09:39.939>easily <00:09:40.113>be <00:09:40.289>plugged <00:09:40.464>in

163
00:09:42.844 --> 00:09:44.924
<00:09:42.844>together <00:09:43.194>with <00:09:43.544>your <00:09:43.894>observability <00:09:44.244>systems.

164
00:09:45.044 --> 00:09:49.004
<00:09:45.044>For <00:09:45.199>example, <00:09:45.355>it <00:09:45.510>works <00:09:45.666>with <00:09:45.821>Langfuse, <00:09:45.977>which <00:09:46.132>we <00:09:46.288>use <00:09:46.444>internally.

165
00:09:50.364 --> 00:09:54.004
<00:09:50.364>It <00:09:50.564>is <00:09:50.764>composed <00:09:50.964>of <00:09:51.164>three <00:09:51.364>main <00:09:51.564>things. <00:09:51.764>Datasets.

166
00:09:54.124 --> 00:09:58.114
<00:09:54.124>These <00:09:54.299>can <00:09:54.474>be <00:09:54.649>generated <00:09:54.824>or <00:09:54.999>curated <00:09:55.174>by <00:09:55.349>yourself <00:09:55.524>and

167
00:09:58.144 --> 00:10:02.134
<00:09:58.144>then <00:09:58.377>uploaded <00:09:58.610>to <00:09:58.844>the <00:09:59.077>platform. <00:09:59.310>We <00:09:59.544>also

168
00:10:02.164 --> 00:10:05.873
<00:10:02.164>have <00:10:02.339>experiments, <00:10:02.514>which <00:10:02.689>are <00:10:02.864>basically <00:10:03.039>pipelines <00:10:03.214>running <00:10:03.389>over <00:10:03.564>your

169
00:10:05.944 --> 00:10:09.204
<00:10:05.944>dataset <00:10:06.294>evaluating <00:10:06.644>a <00:10:06.993>particular <00:10:07.343>endpoint.

170
00:10:09.864 --> 00:10:13.464
<00:10:09.864>And <00:10:10.004>of <00:10:10.144>course, <00:10:10.284>there <00:10:10.424>is <00:10:10.564>metrics <00:10:10.704>defined <00:10:10.844>for <00:10:10.984>RAG <00:10:11.124>and <00:10:11.264>agentic

171
00:10:13.474 --> 00:10:14.424
evaluations,

172
00:10:15.404 --> 00:10:19.184
<00:10:15.404>and <00:10:15.559>these <00:10:15.715>metrics <00:10:15.870>usually <00:10:16.026>use <00:10:16.181>LLM <00:10:16.337>as <00:10:16.492>judge <00:10:16.648>to <00:10:16.804>evaluate

173
00:10:19.224 --> 00:10:19.994
<00:10:19.224>your <00:10:19.917>endpoints.

174
00:10:21.004 --> 00:10:24.504
<00:10:21.004>In <00:10:21.144>today's <00:10:21.284>talk, <00:10:21.424>we'll <00:10:21.564>narrow <00:10:21.704>our <00:10:21.844>focus <00:10:21.984>on <00:10:22.124>the <00:10:22.264>synthetic <00:10:22.404>test

175
00:10:24.604 --> 00:10:28.404
<00:10:24.604>generation <00:10:24.744>using <00:10:24.884>knowledge <00:10:25.024>graphs, <00:10:25.164>and <00:10:25.304>I'll <00:10:25.444>introduce <00:10:25.584>you <00:10:25.724>to <00:10:25.864>the <00:10:26.004>RAG

176
00:10:28.484 --> 00:10:32.264
<00:10:28.484>metrics <00:10:28.834>LLM <00:10:29.184>as <00:10:29.534>judge <00:10:29.884>way.

177
00:10:34.764 --> 00:10:35.804
<00:10:34.764>You <00:10:35.076>might <00:10:35.388>be <00:10:35.699>wondering

178
00:10:36.843 --> 00:10:40.404
<00:10:36.843>why <00:10:37.017>is <00:10:37.193>it <00:10:37.367>required <00:10:37.543>to <00:10:37.718>have <00:10:37.892>a <00:10:38.068>graph <00:10:38.242>to

179
00:10:40.444 --> 00:10:43.184
<00:10:40.444>build <00:10:40.677>a <00:10:40.910>diverse <00:10:41.144>set <00:10:41.377>of <00:10:41.610>test <00:10:41.843>set.

180
00:10:44.184 --> 00:10:48.104
<00:10:44.184>The <00:10:44.311>reason <00:10:44.438>is <00:10:44.565>that <00:10:44.693>when <00:10:44.820>you're <00:10:44.947>building <00:10:45.074>a <00:10:45.202>knowledge <00:10:45.329>graph <00:10:45.456>from <00:10:45.584>your

181
00:10:48.144 --> 00:10:51.744
<00:10:48.144>documents, <00:10:48.319>you <00:10:48.494>have <00:10:48.669>a <00:10:48.844>way <00:10:49.019>to <00:10:49.194>represent <00:10:49.369>chunks <00:10:49.544>as

182
00:10:51.804 --> 00:10:55.584
<00:10:51.804>nodes <00:10:51.944>and <00:10:52.084>then <00:10:52.223>explore <00:10:52.363>how <00:10:52.504>different <00:10:52.644>parts <00:10:52.784>of <00:10:52.924>the <00:10:53.064>documents <00:10:53.204>are

183
00:10:55.624 --> 00:10:56.884
<00:10:55.624>related <00:10:56.002>to <00:10:56.380>one <00:10:56.758>another.

184
00:10:57.924 --> 00:11:01.574
<00:10:57.924>And <00:10:58.051>once <00:10:58.178>you <00:10:58.305>have <00:10:58.433>the <00:10:58.560>knowledge <00:10:58.687>graph, <00:10:58.814>it <00:10:58.942>will <00:10:59.069>represent <00:10:59.196>those <00:10:59.324>links,

185
00:11:01.924 --> 00:11:05.864
<00:11:01.924>and <00:11:02.079>then <00:11:02.235>RAGAS, <00:11:02.390>this <00:11:02.546>framework, <00:11:02.701>can <00:11:02.857>traverse <00:11:03.012>the <00:11:03.168>graph <00:11:03.324>and

186
00:11:05.904 --> 00:11:09.324
<00:11:05.904>then <00:11:06.059>assemble <00:11:06.215>the <00:11:06.370>context <00:11:06.526>that <00:11:06.681>are <00:11:06.837>required <00:11:06.992>to <00:11:07.148>answer <00:11:07.304>specific

187
00:11:09.404 --> 00:11:10.064
questions.

188
00:11:12.364 --> 00:11:16.104
<00:11:12.364>And <00:11:12.471>when <00:11:12.579>you're <00:11:12.687>using <00:11:12.794>RAGAS, <00:11:12.902>you <00:11:13.010>have <00:11:13.117>a <00:11:13.225>way <00:11:13.333>to <00:11:13.440>define <00:11:13.548>what <00:11:13.656>type <00:11:13.764>of

189
00:11:16.144 --> 00:11:18.384
<00:11:16.144>questions <00:11:16.319>you <00:11:16.494>want <00:11:16.669>to <00:11:16.844>have <00:11:17.019>in <00:11:17.194>your <00:11:17.369>test <00:11:17.544>data.

190
00:11:19.124 --> 00:11:19.424
So

191
00:11:20.224 --> 00:11:24.064
<00:11:20.224>the <00:11:20.364>questions <00:11:20.504>range <00:11:20.644>on <00:11:20.784>X, <00:11:20.924>Y <00:11:21.064>axis, <00:11:21.204>so <00:11:21.344>you <00:11:21.484>can <00:11:21.624>define

192
00:11:24.124 --> 00:11:28.094
<00:11:24.124>whether <00:11:24.279>you <00:11:24.435>want <00:11:24.590>specific <00:11:24.746>or <00:11:24.901>abstract <00:11:25.057>questions, <00:11:25.212>whether <00:11:25.368>you <00:11:25.524>have

193
00:11:28.164 --> 00:11:30.354
<00:11:28.164>single-hop <00:11:28.630>or <00:11:29.097>multi-hop <00:11:29.564>questions.

194
00:11:31.324 --> 00:11:34.664
<00:11:31.324>So <00:11:31.464>in <00:11:31.603>the <00:11:31.743>top <00:11:31.883>right <00:11:32.024>corner, <00:11:32.164>you <00:11:32.304>see <00:11:32.444>the <00:11:32.584>specific <00:11:32.723>single-hop,

195
00:11:34.984 --> 00:11:38.944
<00:11:34.984>meaning <00:11:35.084>that <00:11:35.184>you <00:11:35.284>need <00:11:35.384>only <00:11:35.484>one <00:11:35.584>chunk <00:11:35.684>to <00:11:35.784>answer <00:11:35.884>this <00:11:35.984>query, <00:11:36.084>and <00:11:36.184>it <00:11:36.284>is <00:11:36.384>a

196
00:11:39.024 --> 00:11:41.424
<00:11:39.024>very <00:11:39.257>specific <00:11:39.490>question, <00:11:39.724>meaning <00:11:39.957>there <00:11:40.190>is <00:11:40.424>a

197
00:11:41.464 --> 00:11:45.144
<00:11:41.464>one-to-one <00:11:41.814>chunk <00:11:42.164>which <00:11:42.513>contains <00:11:42.863>exactly

198
00:11:45.204 --> 00:11:48.794
<00:11:45.204>the <00:11:45.343>answer <00:11:45.483>to <00:11:45.623>that <00:11:45.763>question. <00:11:45.904>So <00:11:46.044>here <00:11:46.184>the <00:11:46.324>example <00:11:46.464>is <00:11:46.603>what

199
00:11:48.844 --> 00:11:52.754
<00:11:48.844>architecture <00:11:49.194>is <00:11:49.544>a <00:11:49.894>specific <00:11:50.244>tokenizer

200
00:11:53.184 --> 00:11:53.424
for?

201
00:11:55.424 --> 00:11:58.624
<00:11:55.424>You <00:11:55.598>can <00:11:55.774>also <00:11:55.949>define <00:11:56.124>specific <00:11:56.299>questions <00:11:56.473>that <00:11:56.649>are <00:11:56.824>multi-hop.

202
00:11:58.724 --> 00:12:01.564
<00:11:58.724>It <00:11:58.924>means <00:11:59.124>that <00:11:59.324>your <00:11:59.524>RAG <00:11:59.724>system <00:11:59.924>needs <00:12:00.124>to

203
00:12:02.464 --> 00:12:05.964
<00:12:02.464>gather <00:12:02.664>the <00:12:02.864>data <00:12:03.064>to <00:12:03.264>answer <00:12:03.464>that <00:12:03.664>question <00:12:03.864>from

204
00:12:06.024 --> 00:12:07.004
<00:12:06.024>multiple <00:12:06.906>chunks.

205
00:12:08.804 --> 00:12:12.114
<00:12:08.804>You <00:12:08.978>can <00:12:09.154>also <00:12:09.329>define <00:12:09.504>them <00:12:09.679>to <00:12:09.853>be <00:12:10.029>abstract <00:12:10.204>or

206
00:12:12.204 --> 00:12:16.124
<00:12:12.204>specific, <00:12:12.437>meaning, <00:12:12.670>again, <00:12:12.904>if <00:12:13.137>it's <00:12:13.370>abstract, <00:12:13.603>the

207
00:12:16.184 --> 00:12:18.264
<00:12:16.184>LLM <00:12:16.417>needs <00:12:16.650>to <00:12:16.884>do <00:12:17.117>some <00:12:17.350>additional <00:12:17.584>reasoning.

208
00:12:18.364 --> 00:12:22.314
<00:12:18.364>It's <00:12:18.519>not <00:12:18.675>that <00:12:18.830>straightforward <00:12:18.986>to <00:12:19.141>find <00:12:19.297>the <00:12:19.452>answer <00:12:19.608>only <00:12:19.764>from

209
00:12:22.344 --> 00:12:23.844
<00:12:22.344>the <00:12:23.019>documents <00:12:23.694>themselves.

210
00:12:27.374 --> 00:12:31.053
<00:12:27.374>Let's <00:12:27.549>see <00:12:27.724>how <00:12:27.899>RAGAS <00:12:28.074>actually <00:12:28.249>builds <00:12:28.424>that <00:12:28.599>test <00:12:28.774>set.

211
00:12:31.534 --> 00:12:32.674
<00:12:31.534>In <00:12:31.876>the <00:12:32.218>first <00:12:32.560>step,

212
00:12:34.054 --> 00:12:37.874
<00:12:34.054>we <00:12:34.181>are <00:12:34.308>going <00:12:34.435>to <00:12:34.563>go <00:12:34.690>ahead <00:12:34.817>and <00:12:34.944>split <00:12:35.072>the <00:12:35.199>documents <00:12:35.326>into <00:12:35.454>chunks.

213
00:12:38.214 --> 00:12:40.674
<00:12:38.214>Later <00:12:38.414>on, <00:12:38.613>these <00:12:38.814>will <00:12:39.013>be <00:12:39.214>called <00:12:39.414>the <00:12:39.613>nodes.

214
00:12:42.034 --> 00:12:45.254
<00:12:42.034>Once <00:12:42.189>we <00:12:42.345>have <00:12:42.500>the <00:12:42.656>nodes, <00:12:42.811>we <00:12:42.967>do <00:12:43.122>some <00:12:43.278>extraction <00:12:43.434>with

215
00:12:45.294 --> 00:12:48.714
<00:12:45.294>synthesizers, <00:12:45.494>where <00:12:45.694>we <00:12:45.894>extract <00:12:46.093>a <00:12:46.294>variety <00:12:46.494>of <00:12:46.694>things.

216
00:12:49.194 --> 00:12:52.944
<00:12:49.194>You <00:12:49.427>can <00:12:49.660>extract <00:12:49.894>named <00:12:50.127>entities, <00:12:50.360>themes, <00:12:50.593>and

217
00:12:53.074 --> 00:12:54.934
<00:12:53.074>summaries <00:12:53.307>that <00:12:53.540>are <00:12:53.774>parts <00:12:54.007>of <00:12:54.240>your <00:12:54.473>chunks.

218
00:12:55.934 --> 00:12:59.894
<00:12:55.934>In <00:12:56.050>the <00:12:56.167>next <00:12:56.284>step, <00:12:56.400>you <00:12:56.517>will <00:12:56.634>try <00:12:56.750>to <00:12:56.867>find <00:12:56.983>relationships <00:12:57.100>between <00:12:57.217>the <00:12:57.334>nodes,

219
00:13:00.294 --> 00:13:04.214
<00:13:00.294>checking <00:13:00.449>whether <00:13:00.605>there <00:13:00.760>are <00:13:00.916>any <00:13:01.071>shared <00:13:01.227>themes, <00:13:01.382>entities, <00:13:01.538>or <00:13:01.694>summaries.

220
00:13:04.914 --> 00:13:07.874
<00:13:04.914>If <00:13:05.054>there <00:13:05.194>are, <00:13:05.334>then <00:13:05.473>we <00:13:05.614>will <00:13:05.754>draw <00:13:05.894>an <00:13:06.034>edge <00:13:06.174>between <00:13:06.314>those

221
00:13:07.934 --> 00:13:11.724
<00:13:07.934>nodes. <00:13:08.134>And <00:13:08.334>this <00:13:08.534>whole <00:13:08.733>process <00:13:08.934>is <00:13:09.134>the <00:13:09.334>transformation.

222
00:13:11.814 --> 00:13:14.734
<00:13:11.814>By <00:13:11.954>the <00:13:12.093>end <00:13:12.233>of <00:13:12.373>that, <00:13:12.514>you'll <00:13:12.654>have <00:13:12.794>the <00:13:12.934>knowledge <00:13:13.074>graph <00:13:13.214>available.

223
00:13:15.854 --> 00:13:17.674
<00:13:15.854>So <00:13:16.029>this <00:13:16.204>was <00:13:16.379>the <00:13:16.554>first <00:13:16.729>step <00:13:16.904>of <00:13:17.079>this <00:13:17.254>flow,

224
00:13:18.554 --> 00:13:22.234
<00:13:18.554>and <00:13:18.654>in <00:13:18.754>the <00:13:18.853>next <00:13:18.954>step, <00:13:19.054>we're <00:13:19.154>going <00:13:19.254>to <00:13:19.353>generate <00:13:19.454>questions <00:13:19.554>and <00:13:19.654>draw <00:13:19.754>a <00:13:19.853>couple <00:13:19.954>of

225
00:13:22.294 --> 00:13:26.074
<00:13:22.294>scenarios. <00:13:22.494>So <00:13:22.694>when <00:13:22.894>you're <00:13:23.093>using <00:13:23.294>RAGAS, <00:13:23.494>you <00:13:23.694>can

226
00:13:26.134 --> 00:13:27.634
<00:13:26.134>define <00:13:26.809>the <00:13:27.484>persona,

227
00:13:28.994 --> 00:13:31.713
<00:13:28.994>the <00:13:29.169>typical <00:13:29.344>user <00:13:29.519>that <00:13:29.694>will <00:13:29.869>interact <00:13:30.044>with <00:13:30.219>your <00:13:30.394>system.

228
00:13:32.274 --> 00:13:35.794
<00:13:32.274>You <00:13:32.367>can <00:13:32.460>set <00:13:32.554>up <00:13:32.647>how <00:13:32.740>long <00:13:32.834>the <00:13:32.927>questions <00:13:33.020>should <00:13:33.114>be <00:13:33.207>and <00:13:33.300>what <00:13:33.394>style <00:13:33.487>they <00:13:33.580>should <00:13:33.674>be

229
00:13:35.854 --> 00:13:39.554
<00:13:35.854>using, <00:13:36.087>and <00:13:36.320>which <00:13:36.554>of <00:13:36.787>the <00:13:37.020>query <00:13:37.254>synthesizers

230
00:13:39.854 --> 00:13:42.634
<00:13:39.854>it <00:13:40.054>should <00:13:40.254>rely <00:13:40.454>on <00:13:40.654>to <00:13:40.854>generate <00:13:41.054>the <00:13:41.254>scenarios.

231
00:13:42.714 --> 00:13:46.704
<00:13:42.714>So <00:13:42.869>for <00:13:43.025>example, <00:13:43.180>I <00:13:43.336>want <00:13:43.491>to <00:13:43.647>focus <00:13:43.802>on <00:13:43.958>nodes <00:13:44.113>that

232
00:13:46.754 --> 00:13:50.654
<00:13:46.754>have <00:13:46.987>similar <00:13:47.220>entities <00:13:47.454>in <00:13:47.687>them. <00:13:47.920>And <00:13:48.154>once

233
00:13:50.714 --> 00:13:54.494
<00:13:50.714>these <00:13:50.914>synthesizers <00:13:51.113>run, <00:13:51.314>you <00:13:51.513>have <00:13:51.714>the <00:13:51.914>output <00:13:52.113>test

234
00:13:54.574 --> 00:13:58.094
<00:13:54.574>set, <00:13:54.748>and <00:13:54.924>then <00:13:55.098>in <00:13:55.274>here <00:13:55.449>you <00:13:55.623>have <00:13:55.799>separate <00:13:55.973>rows.

235
00:13:58.754 --> 00:14:01.334
<00:13:58.754>Each <00:13:58.909>of <00:13:59.065>the <00:13:59.220>row <00:13:59.376>is <00:13:59.531>going <00:13:59.687>to <00:13:59.842>have <00:13:59.998>the <00:14:00.154>question,

236
00:14:02.134 --> 00:14:05.914
<00:14:02.134>the <00:14:02.250>gold <00:14:02.367>chunks <00:14:02.484>that <00:14:02.600>were <00:14:02.717>used <00:14:02.834>to <00:14:02.950>answer <00:14:03.067>that <00:14:03.184>question, <00:14:03.300>and <00:14:03.417>also <00:14:03.534>the

237
00:14:05.954 --> 00:14:07.464
<00:14:05.954>gold <00:14:06.293>answer, <00:14:06.633>the <00:14:06.973>label <00:14:07.313>there.

238
00:14:08.994 --> 00:14:12.504
<00:14:08.994>What <00:14:09.087>I <00:14:09.180>do <00:14:09.274>recommend <00:14:09.367>is <00:14:09.460>to <00:14:09.554>play <00:14:09.647>around <00:14:09.740>with <00:14:09.834>this <00:14:09.927>a <00:14:10.020>little <00:14:10.114>bit <00:14:10.207>so <00:14:10.300>you <00:14:10.394>can

239
00:14:12.514 --> 00:14:16.394
<00:14:12.514>define <00:14:12.714>different <00:14:12.914>personas. <00:14:13.114>You <00:14:13.314>can <00:14:13.514>also <00:14:13.714>weigh <00:14:13.914>the

240
00:14:16.454 --> 00:14:20.274
<00:14:16.454>distribution <00:14:16.654>between <00:14:16.853>single-hop <00:14:17.054>and <00:14:17.253>multi-hop <00:14:17.454>questions, <00:14:17.654>and <00:14:17.853>also

241
00:14:20.334 --> 00:14:24.294
<00:14:20.334>which <00:14:20.473>synthesizers <00:14:20.613>are <00:14:20.753>used <00:14:20.893>for <00:14:21.034>each <00:14:21.174>of <00:14:21.314>the <00:14:21.454>questions <00:14:21.593>so <00:14:21.733>that

242
00:14:24.374 --> 00:14:28.294
<00:14:24.374>it <00:14:24.549>better <00:14:24.724>resembles <00:14:24.899>the <00:14:25.074>user <00:14:25.249>interactions <00:14:25.424>that <00:14:25.599>you're <00:14:25.774>likely

243
00:14:28.354 --> 00:14:30.554
<00:14:28.354>to <00:14:28.820>anticipate <00:14:29.287>later <00:14:29.754>on.

244
00:14:32.874 --> 00:14:36.474
<00:14:32.874>Enough <00:14:33.049>about <00:14:33.224>the <00:14:33.399>theory <00:14:33.574>now. <00:14:33.749>Let's <00:14:33.924>jump <00:14:34.099>into <00:14:34.274>the

245
00:14:36.574 --> 00:14:36.894
code.

246
00:14:43.174 --> 00:14:44.334
<00:14:43.174>So <00:14:43.522>as <00:14:43.870>I've <00:14:44.218>mentioned,

247
00:14:45.414 --> 00:14:47.934
<00:14:45.414>I <00:14:45.589>have <00:14:45.764>this <00:14:45.939>notebook <00:14:46.114>available <00:14:46.289>for <00:14:46.464>you <00:14:46.639>on <00:14:46.814>GitHub,

248
00:14:48.814 --> 00:14:52.614
<00:14:48.814>and <00:14:48.930>in <00:14:49.047>here <00:14:49.164>I've <00:14:49.280>already <00:14:49.397>pre-run <00:14:49.514>this <00:14:49.630>notebook <00:14:49.747>so <00:14:49.863>that <00:14:49.980>we <00:14:50.097>don't <00:14:50.214>waste

249
00:14:52.694 --> 00:14:55.494
<00:14:52.694>time <00:14:52.868>on <00:14:53.044>generating <00:14:53.218>the <00:14:53.394>knowledge <00:14:53.569>graph <00:14:53.743>and <00:14:53.919>so <00:14:54.093>on.

250
00:14:56.334 --> 00:15:00.274
<00:14:56.334>In <00:14:56.450>the <00:14:56.567>first <00:14:56.684>setup, <00:14:56.800>we <00:14:56.917>are <00:14:57.034>importing <00:14:57.150>a <00:14:57.267>couple <00:14:57.384>of <00:14:57.500>libraries, <00:14:57.617>RAGAS, <00:14:57.734>and

251
00:15:00.354 --> 00:15:03.514
<00:15:00.354>also <00:15:00.494>I <00:15:00.634>need <00:15:00.774>you <00:15:00.914>to <00:15:01.054>define <00:15:01.194>an <00:15:01.334>OpenAI <00:15:01.474>API <00:15:01.614>key <00:15:01.754>in

252
00:15:03.614 --> 00:15:07.354
<00:15:03.614>your <00:15:03.814>.env <00:15:04.014>file <00:15:04.214>because <00:15:04.414>we'll <00:15:04.614>use <00:15:04.814>OpenAI <00:15:05.014>for

253
00:15:07.474 --> 00:15:11.114
<00:15:07.474>this <00:15:07.707>experiment, <00:15:07.940>and <00:15:08.174>we <00:15:08.407>are <00:15:08.640>using <00:15:08.874>0.4.3

254
00:15:11.234 --> 00:15:15.194
<00:15:11.234>of <00:15:11.374>the <00:15:11.514>RAGAS <00:15:11.654>library. <00:15:11.794>In <00:15:11.934>the <00:15:12.074>next <00:15:12.214>step, <00:15:12.354>we <00:15:12.494>are <00:15:12.634>setting

255
00:15:15.234 --> 00:15:18.154
<00:15:15.234>up <00:15:15.434>a <00:15:15.634>very <00:15:15.834>simple <00:15:16.034>RAG <00:15:16.234>system, <00:15:16.434>and <00:15:16.634>we

256
00:15:18.234 --> 00:15:22.084
<00:15:18.234>are <00:15:18.389>deliberately <00:15:18.545>making <00:15:18.700>it <00:15:18.856>very <00:15:19.011>simple <00:15:19.167>for <00:15:19.322>the <00:15:19.478>sake <00:15:19.634>of

257
00:15:22.154 --> 00:15:23.194
<00:15:22.154>this <00:15:23.089>demonstration.

258
00:15:25.134 --> 00:15:26.874
<00:15:25.134>Here, <00:15:25.834>I'm <00:15:26.534>using

259
00:15:27.754 --> 00:15:31.454
<00:15:27.754>a <00:15:27.881>data <00:15:28.008>set. <00:15:28.135>This <00:15:28.263>is <00:15:28.390>the <00:15:28.517>Hugging <00:15:28.644>Face <00:15:28.772>doc <00:15:28.899>data <00:15:29.026>set, <00:15:29.154>publicly

260
00:15:31.494 --> 00:15:35.374
<00:15:31.494>accessible, <00:15:31.634>and <00:15:31.774>I'm <00:15:31.914>only <00:15:32.054>loading <00:15:32.194>50 <00:15:32.334>documents <00:15:32.474>to <00:15:32.614>keep <00:15:32.754>this <00:15:32.894>small

261
00:15:35.414 --> 00:15:35.894
<00:15:35.414>and <00:15:35.846>quick.

262
00:15:37.854 --> 00:15:41.314
<00:15:37.854>And <00:15:37.970>let's <00:15:38.087>actually <00:15:38.204>have <00:15:38.320>a <00:15:38.437>look <00:15:38.554>at <00:15:38.670>how <00:15:38.787>this <00:15:38.904>data <00:15:39.020>set <00:15:39.137>looks <00:15:39.254>like.

263
00:15:43.794 --> 00:15:45.754
<00:15:43.794>So <00:15:44.027>right <00:15:44.260>here, <00:15:44.494>if <00:15:44.727>you <00:15:44.960>come <00:15:45.194>to

264
00:15:47.394 --> 00:15:51.344
<00:15:47.394>Hugging <00:15:47.569>Face, <00:15:47.744>you'll <00:15:47.919>see <00:15:48.094>that <00:15:48.269>this <00:15:48.444>data <00:15:48.619>set <00:15:48.794>has

265
00:15:51.394 --> 00:15:55.364
<00:15:51.394>two <00:15:51.549>columns, <00:15:51.705>the <00:15:51.860>text <00:15:52.016>itself, <00:15:52.171>which <00:15:52.327>is <00:15:52.482>in <00:15:52.638>markdown, <00:15:52.794>and

266
00:15:55.394 --> 00:15:58.354
<00:15:55.394>the <00:15:55.569>source, <00:15:55.744>the <00:15:55.919>link <00:15:56.094>where <00:15:56.269>this <00:15:56.444>text <00:15:56.619>came <00:15:56.794>from.

267
00:15:58.434 --> 00:16:02.414
<00:15:58.434>So <00:15:58.550>we <00:15:58.667>can <00:15:58.784>just <00:15:58.900>click <00:15:59.017>into <00:15:59.134>one <00:15:59.250>of <00:15:59.367>the <00:15:59.483>examples <00:15:59.600>right <00:15:59.717>here <00:15:59.834>to

268
00:16:02.474 --> 00:16:03.443
<00:16:02.474>see <00:16:02.692>how <00:16:02.910>it <00:16:03.128>looks <00:16:03.346>like.

269
00:16:04.454 --> 00:16:06.654
<00:16:04.454>We <00:16:04.733>see <00:16:05.013>some <00:16:05.294>markdown <00:16:05.574>formatted <00:16:05.853>text,

270
00:16:07.794 --> 00:16:10.014
<00:16:07.794>and <00:16:07.994>then <00:16:08.194>where <00:16:08.394>it <00:16:08.593>came <00:16:08.794>from <00:16:08.994>as <00:16:09.194>well.

271
00:16:14.534 --> 00:16:15.754
<00:16:14.534>Once <00:16:14.808>we <00:16:15.083>have <00:16:15.357>loaded <00:16:15.632>that,

272
00:16:16.814 --> 00:16:20.494
<00:16:16.814>we'll <00:16:17.014>just <00:16:17.214>go <00:16:17.414>ahead <00:16:17.613>and <00:16:17.814>define <00:16:18.014>our <00:16:18.214>models.

273
00:16:20.574 --> 00:16:24.134
<00:16:20.574>We <00:16:20.729>are <00:16:20.885>using <00:16:21.040>an <00:16:21.196>embedding <00:16:21.351>model <00:16:21.507>and <00:16:21.662>a <00:16:21.818>GPT-4o <00:16:21.973>mini

274
00:16:24.754 --> 00:16:27.174
<00:16:24.754>as <00:16:24.987>our <00:16:25.220>main <00:16:25.454>LLM <00:16:25.687>for <00:16:25.920>this <00:16:26.154>experiment.

275
00:16:27.414 --> 00:16:31.094
<00:16:27.414>In <00:16:27.614>the <00:16:27.814>first <00:16:28.014>step, <00:16:28.214>we'll <00:16:28.414>split <00:16:28.614>the <00:16:28.814>data

276
00:16:31.194 --> 00:16:35.154
<00:16:31.194>using <00:16:31.349>LangChain's <00:16:31.505>recursive <00:16:31.660>character <00:16:31.816>text <00:16:31.971>splitter <00:16:32.127>with <00:16:32.282>a <00:16:32.438>chunk <00:16:32.593>size

277
00:16:35.194 --> 00:16:38.734
<00:16:35.194>of <00:16:35.368>1,000 <00:16:35.544>characters <00:16:35.718>and <00:16:35.894>using <00:16:36.069>an <00:16:36.243>overlap <00:16:36.419>of <00:16:36.593>100

278
00:16:38.814 --> 00:16:39.394
characters.

279
00:16:40.574 --> 00:16:43.334
<00:16:40.574>Once <00:16:40.774>we <00:16:40.973>have <00:16:41.174>the <00:16:41.373>chunks <00:16:41.574>available, <00:16:41.774>which <00:16:41.973>are

280
00:16:43.534 --> 00:16:46.754
<00:16:43.534>462 <00:16:43.767>of <00:16:44.000>them, <00:16:44.234>we'll <00:16:44.467>just <00:16:44.700>go <00:16:44.934>ahead

281
00:16:46.954 --> 00:16:50.654
<00:16:46.954>and <00:16:47.233>embed <00:16:47.513>them <00:16:47.794>using <00:16:48.074>the <00:16:48.353>embedding

282
00:16:50.674 --> 00:16:52.174
<00:16:50.674>model <00:16:51.124>defined <00:16:51.574>up <00:16:52.024>there.

283
00:16:54.034 --> 00:16:57.724
<00:16:54.034>Then <00:16:54.209>we'll <00:16:54.384>continue <00:16:54.559>and <00:16:54.734>define <00:16:54.909>a <00:16:55.084>retrieval <00:16:55.259>function <00:16:55.434>and

284
00:16:58.314 --> 00:17:02.224
<00:16:58.314>also <00:16:58.454>the <00:16:58.593>function <00:16:58.733>for <00:16:58.873>RAG <00:16:59.014>to <00:16:59.154>work, <00:16:59.294>defining <00:16:59.434>the <00:16:59.574>system <00:16:59.714>prompt

285
00:17:02.354 --> 00:17:06.295
<00:17:02.354>and <00:17:02.461>the <00:17:02.569>user <00:17:02.677>prompt <00:17:02.784>as <00:17:02.892>well. <00:17:03.000>And <00:17:03.107>to <00:17:03.215>make <00:17:03.323>sure <00:17:03.430>everything <00:17:03.538>works, <00:17:03.646>I'm <00:17:03.754>just

286
00:17:06.305 --> 00:17:09.714
<00:17:06.305>sending <00:17:06.460>a <00:17:06.616>query, <00:17:06.771>"What's <00:17:06.927>a <00:17:07.082>tokenizer <00:17:07.238>in <00:17:07.393>a <00:17:07.549>single <00:17:07.705>sentence?"

287
00:17:10.454 --> 00:17:13.974
<00:17:10.454>And <00:17:10.570>then <00:17:10.687>in <00:17:10.803>three <00:17:10.920>seconds, <00:17:11.037>I <00:17:11.154>get <00:17:11.270>the <00:17:11.387>answer <00:17:11.503>back <00:17:11.620>about <00:17:11.737>what <00:17:11.854>a

288
00:17:14.075 --> 00:17:15.154
<00:17:14.075>tokenizer <00:17:15.046>is.

289
00:17:17.555 --> 00:17:21.295
<00:17:17.555>So <00:17:17.755>let's <00:17:17.955>continue <00:17:18.155>to <00:17:18.355>generating <00:17:18.555>the <00:17:18.755>synthetic <00:17:18.955>test

290
00:17:21.335 --> 00:17:22.854
<00:17:21.335>set <00:17:22.018>with <00:17:22.702>RAGAS.

291
00:17:24.555 --> 00:17:28.154
<00:17:24.555>Here, <00:17:24.695>we <00:17:24.835>are <00:17:24.975>defining <00:17:25.115>the <00:17:25.255>generator <00:17:25.395>with <00:17:25.535>the <00:17:25.675>model <00:17:25.815>and <00:17:25.955>the

292
00:17:28.194 --> 00:17:29.374
<00:17:28.194>embedding <00:17:28.548>we <00:17:28.902>are <00:17:29.256>using.

293
00:17:30.794 --> 00:17:33.214
<00:17:30.794>Right <00:17:30.994>here, <00:17:31.194>we <00:17:31.394>are <00:17:31.594>building <00:17:31.794>the <00:17:31.994>knowledge <00:17:32.194>graph.

294
00:17:34.994 --> 00:17:38.374
<00:17:34.994>And <00:17:35.149>then <00:17:35.305>this <00:17:35.460>apply_transforms <00:17:35.616>function <00:17:35.771>is <00:17:35.927>the <00:17:36.082>one <00:17:36.238>that <00:17:36.394>will

295
00:17:38.414 --> 00:17:42.334
<00:17:38.414>actually <00:17:38.614>create <00:17:38.814>all <00:17:39.014>those <00:17:39.214>transformations, <00:17:39.414>extracting <00:17:39.614>the <00:17:39.814>entities,

296
00:17:42.434 --> 00:17:43.974
<00:17:42.434>themes, <00:17:42.711>and <00:17:42.988>the <00:17:43.265>summaries <00:17:43.542>from <00:17:43.820>them.

297
00:17:46.184 --> 00:17:49.874
<00:17:46.184>And <00:17:46.300>once <00:17:46.417>we <00:17:46.533>have <00:17:46.650>the <00:17:46.767>knowledge <00:17:46.884>graph, <00:17:47.000>we <00:17:47.117>can <00:17:47.234>traverse <00:17:47.350>the <00:17:47.467>knowledge <00:17:47.584>graph

298
00:17:49.974 --> 00:17:53.254
<00:17:49.974>to <00:17:50.129>synthesize <00:17:50.285>the <00:17:50.440>queries <00:17:50.596>and <00:17:50.751>then <00:17:50.907>generate <00:17:51.062>the <00:17:51.218>test <00:17:51.374>set.

299
00:17:53.334 --> 00:17:56.974
<00:17:53.334>In <00:17:53.441>this <00:17:53.549>case, <00:17:53.657>I <00:17:53.764>just <00:17:53.872>wanted <00:17:53.980>eight <00:17:54.087>simple <00:17:54.195>questions <00:17:54.303>to <00:17:54.410>come <00:17:54.518>out <00:17:54.626>of <00:17:54.734>that

300
00:17:58.134 --> 00:18:01.974
<00:17:58.134>and <00:17:58.250>have <00:17:58.367>that <00:17:58.484>available. <00:17:58.600>This <00:17:58.717>can <00:17:58.834>take <00:17:58.950>some <00:17:59.067>time <00:17:59.184>for <00:17:59.300>all <00:17:59.417>of <00:17:59.534>the

301
00:18:02.014 --> 00:18:03.374
<00:18:02.014>extractors <00:18:02.422>to <00:18:02.830>run <00:18:03.238>through.

302
00:18:05.814 --> 00:18:08.354
<00:18:05.814>And <00:18:05.969>then <00:18:06.125>once <00:18:06.280>that's <00:18:06.436>done, <00:18:06.591>we <00:18:06.747>have <00:18:06.902>a <00:18:07.058>data <00:18:07.214>frame.

303
00:18:08.594 --> 00:18:12.454
<00:18:08.594>So <00:18:08.694>let's <00:18:08.794>see <00:18:08.894>what <00:18:08.994>is <00:18:09.094>in <00:18:09.194>there. <00:18:09.294>For <00:18:09.394>our <00:18:09.494>test <00:18:09.594>set, <00:18:09.694>we <00:18:09.794>have <00:18:09.894>eight <00:18:09.994>rows,

304
00:18:12.554 --> 00:18:14.434
<00:18:12.554>meaning <00:18:12.904>eight <00:18:13.254>questions <00:18:13.604>were <00:18:13.954>generated.

305
00:18:15.774 --> 00:18:19.583
<00:18:15.774>And <00:18:15.890>for <00:18:16.007>each <00:18:16.123>one <00:18:16.240>of <00:18:16.357>the <00:18:16.474>questions, <00:18:16.590>there <00:18:16.707>is <00:18:16.823>a <00:18:16.940>reference <00:18:17.057>that <00:18:17.174>is

306
00:18:19.614 --> 00:18:23.434
<00:18:19.614>the <00:18:19.847>answer <00:18:20.080>that <00:18:20.314>was <00:18:20.547>generated. <00:18:20.780>The <00:18:21.014>reference

307
00:18:23.534 --> 00:18:26.034
<00:18:23.534>contexts <00:18:23.767>are <00:18:24.000>the <00:18:24.234>chunks <00:18:24.467>that <00:18:24.700>were <00:18:24.934>used.

308
00:18:26.764 --> 00:18:30.714
<00:18:26.764>And <00:18:26.997>the <00:18:27.230>synthesizer <00:18:27.464>here <00:18:27.697>defines <00:18:27.930>whether <00:18:28.164>a

309
00:18:30.754 --> 00:18:33.214
<00:18:30.754>single-hop <00:18:31.103>or <00:18:31.454>multi-hop <00:18:31.803>was <00:18:32.154>used

310
00:18:34.274 --> 00:18:37.054
<00:18:34.274>And <00:18:34.474>we <00:18:34.674>can <00:18:34.873>see <00:18:35.073>an <00:18:35.274>example <00:18:35.474>as <00:18:35.674>well.

311
00:18:39.014 --> 00:18:40.424
<00:18:39.014>Here <00:18:39.331>you <00:18:39.648>see <00:18:39.965>the <00:18:40.283>question,

312
00:18:41.614 --> 00:18:42.974
<00:18:41.614>you <00:18:41.920>see <00:18:42.226>the <00:18:42.532>reference <00:18:42.838>answer,

313
00:18:43.894 --> 00:18:47.494
<00:18:43.894>and <00:18:44.034>you <00:18:44.174>can <00:18:44.314>also <00:18:44.454>see <00:18:44.594>that <00:18:44.734>two <00:18:44.874>chunks <00:18:45.014>were <00:18:45.154>used <00:18:45.294>to

314
00:18:47.534 --> 00:18:50.704
<00:18:47.534>generate <00:18:47.709>that <00:18:47.884>answer. <00:18:48.059>So <00:18:48.234>this <00:18:48.409>was <00:18:48.584>a <00:18:48.759>multi-hop <00:18:48.934>query,

315
00:18:51.954 --> 00:18:55.654
<00:18:51.954>abstract <00:18:52.420>or <00:18:52.887>specific. <00:18:53.354>Great.

316
00:18:55.774 --> 00:18:59.074
<00:18:55.774>So <00:18:55.929>what <00:18:56.085>I <00:18:56.240>did <00:18:56.396>here, <00:18:56.551>I <00:18:56.707>just <00:18:56.862>went <00:18:57.018>ahead <00:18:57.174>and

317
00:19:00.494 --> 00:19:04.314
<00:19:00.494>saved <00:19:00.621>the <00:19:00.748>knowledge <00:19:00.875>graph <00:19:01.003>as <00:19:01.130>a <00:19:01.257>JSON <00:19:01.384>and <00:19:01.512>then <00:19:01.639>the <00:19:01.766>nice <00:19:01.894>thing

318
00:19:04.414 --> 00:19:07.894
<00:19:04.414>is <00:19:04.589>that <00:19:04.764>you <00:19:04.939>can <00:19:05.114>use <00:19:05.289>just <00:19:05.464>some <00:19:05.639>simple <00:19:05.814>MATLAB

319
00:19:08.214 --> 00:19:12.034
<00:19:08.214>visualization <00:19:08.389>to <00:19:08.563>have <00:19:08.739>a <00:19:08.914>look <00:19:09.089>at <00:19:09.264>how <00:19:09.438>this <00:19:09.614>knowledge

320
00:19:12.174 --> 00:19:13.634
<00:19:12.174>graph <00:19:12.612>actually <00:19:13.050>looks <00:19:13.488>like.

321
00:19:15.294 --> 00:19:19.114
<00:19:15.294>So <00:19:15.401>if <00:19:15.509>we <00:19:15.617>jump <00:19:15.724>right <00:19:15.832>in <00:19:15.940>here, <00:19:16.047>we <00:19:16.155>see <00:19:16.263>that <00:19:16.370>there <00:19:16.478>is <00:19:16.586>the <00:19:16.694>462

322
00:19:19.194 --> 00:19:21.714
<00:19:19.194>nodes, <00:19:19.474>which <00:19:19.754>are <00:19:20.033>the <00:19:20.313>chunks <00:19:20.594>extracted.

323
00:19:22.634 --> 00:19:26.414
<00:19:22.634>And <00:19:22.734>if <00:19:22.834>we <00:19:22.934>zoom <00:19:23.034>in <00:19:23.134>and <00:19:23.234>if <00:19:23.334>we <00:19:23.434>hover <00:19:23.534>over <00:19:23.634>each <00:19:23.734>of <00:19:23.834>the <00:19:23.934>nodes, <00:19:24.034>you'll

324
00:19:26.534 --> 00:19:30.274
<00:19:26.534>see <00:19:26.884>which <00:19:27.234>exact <00:19:27.584>text <00:19:27.934>snippet

325
00:19:30.494 --> 00:19:31.074
<00:19:30.494>it <00:19:31.016>was.

326
00:19:32.314 --> 00:19:35.734
<00:19:32.314>And <00:19:32.454>these <00:19:32.594>are <00:19:32.734>the <00:19:32.874>edges, <00:19:33.014>based <00:19:33.154>on <00:19:33.294>which <00:19:33.434>they <00:19:33.574>are <00:19:33.714>related.

327
00:19:35.834 --> 00:19:39.364
<00:19:35.834>So <00:19:36.009>you <00:19:36.184>see <00:19:36.359>that <00:19:36.534>between <00:19:36.709>these <00:19:36.884>two <00:19:37.059>chunks, <00:19:37.234>some

328
00:19:39.834 --> 00:19:41.014
<00:19:39.834>entities <00:19:40.896>overlap.

329
00:19:42.014 --> 00:19:45.304
<00:19:42.014>You <00:19:42.188>can <00:19:42.363>also <00:19:42.539>find <00:19:42.714>examples <00:19:42.889>where <00:19:43.063>they <00:19:43.238>have <00:19:43.414>similar

330
00:19:47.054 --> 00:19:47.574
summaries.

331
00:19:48.394 --> 00:19:52.134
<00:19:48.394>And <00:19:48.534>what <00:19:48.674>I <00:19:48.814>find <00:19:48.954>interesting <00:19:49.094>is <00:19:49.234>you <00:19:49.374>can <00:19:49.514>also <00:19:49.654>see <00:19:49.794>that

332
00:19:52.254 --> 00:19:55.854
<00:19:52.254>some <00:19:52.454>chunks <00:19:52.654>were <00:19:52.853>highly <00:19:53.053>related, <00:19:53.254>and <00:19:53.454>they <00:19:53.654>are

333
00:19:56.074 --> 00:19:59.454
<00:19:56.074>grouped <00:19:56.201>together <00:19:56.328>very <00:19:56.455>closely. <00:19:56.583>But <00:19:56.710>down <00:19:56.837>here <00:19:56.964>you <00:19:57.092>can <00:19:57.219>also <00:19:57.346>see <00:19:57.474>the

334
00:19:59.534 --> 00:20:03.054
<00:19:59.534>isolated <00:19:59.734>ones <00:19:59.934>where <00:20:00.134>there <00:20:00.334>were <00:20:00.534>no <00:20:00.734>similarities <00:20:00.934>found.

335
00:20:03.094 --> 00:20:05.714
<00:20:03.094>So <00:20:03.294>they <00:20:03.494>are <00:20:03.694>very <00:20:03.894>unique <00:20:04.094>in <00:20:04.294>this <00:20:04.494>database.

336
00:20:10.214 --> 00:20:10.754
Great.

337
00:20:13.234 --> 00:20:17.084
<00:20:13.234>And <00:20:13.350>down <00:20:13.467>here, <00:20:13.583>I <00:20:13.700>just <00:20:13.817>wanted <00:20:13.934>to <00:20:14.050>show <00:20:14.167>you <00:20:14.283>how <00:20:14.400>you <00:20:14.517>can <00:20:14.634>define

338
00:20:17.114 --> 00:20:19.514
<00:20:17.114>the <00:20:17.394>particular <00:20:17.674>personas <00:20:17.954>for <00:20:18.234>your <00:20:18.514>experiments.

339
00:20:19.594 --> 00:20:20.244
<00:20:19.594>So <00:20:19.886>right <00:20:20.178>here,

340
00:20:21.254 --> 00:20:25.134
<00:20:21.254>I <00:20:21.353>can <00:20:21.454>define <00:20:21.553>who <00:20:21.654>will <00:20:21.754>be <00:20:21.853>the <00:20:21.954>user <00:20:22.053>and <00:20:22.154>what <00:20:22.254>kind <00:20:22.353>of <00:20:22.454>characteristics <00:20:22.553>they <00:20:22.654>might

341
00:20:25.194 --> 00:20:28.854
<00:20:25.194>have. <00:20:25.427>You <00:20:25.660>can <00:20:25.894>also <00:20:26.127>have <00:20:26.360>the <00:20:26.594>freedom

342
00:20:28.914 --> 00:20:32.014
<00:20:28.914>to <00:20:29.147>define <00:20:29.380>the <00:20:29.614>distribution <00:20:29.847>between <00:20:30.080>the <00:20:30.314>different

343
00:20:32.074 --> 00:20:35.444
<00:20:32.074>synthesizers <00:20:32.274>and <00:20:32.474>what <00:20:32.674>properties <00:20:32.874>they <00:20:33.074>should <00:20:33.274>be <00:20:33.474>focusing

344
00:20:35.514 --> 00:20:39.274
<00:20:35.514>on. <00:20:35.654>These <00:20:35.793>numbers <00:20:35.934>here <00:20:36.073>add <00:20:36.214>up <00:20:36.353>to <00:20:36.494>100, <00:20:36.633>where <00:20:36.774>you <00:20:36.914>can

345
00:20:39.294 --> 00:20:42.134
<00:20:39.294>define <00:20:39.527>the <00:20:39.760>percentages <00:20:39.994>of <00:20:40.227>each <00:20:40.460>query <00:20:40.694>type.

346
00:20:45.394 --> 00:20:49.294
<00:20:45.394>So <00:20:45.521>once <00:20:45.648>we <00:20:45.775>have <00:20:45.903>our <00:20:46.030>test <00:20:46.157>set <00:20:46.284>generated, <00:20:46.412>let's <00:20:46.539>just <00:20:46.666>come <00:20:46.794>back

347
00:20:49.474 --> 00:20:51.194
<00:20:49.474>to <00:20:49.940>the <00:20:50.407>next <00:20:50.874>steps.

348
00:20:55.834 --> 00:20:59.574
<00:20:55.834>So <00:20:55.989>once <00:20:56.145>we <00:20:56.300>have <00:20:56.456>this, <00:20:56.611>of <00:20:56.767>course, <00:20:56.922>we <00:20:57.078>want <00:20:57.234>to

349
00:20:59.634 --> 00:21:03.534
<00:20:59.634>evaluate <00:20:59.761>the <00:20:59.888>RAG <00:21:00.015>system, <00:21:00.143>and <00:21:00.270>for <00:21:00.397>that <00:21:00.524>there <00:21:00.652>will <00:21:00.779>be <00:21:00.906>different <00:21:01.034>metrics

350
00:21:03.634 --> 00:21:03.954
used.

351
00:21:05.314 --> 00:21:09.064
<00:21:05.314>Ragas <00:21:05.489>offers <00:21:05.664>a <00:21:05.839>variety <00:21:06.014>of <00:21:06.189>metrics <00:21:06.364>over <00:21:06.539>here, <00:21:06.714>and

352
00:21:09.154 --> 00:21:12.834
<00:21:09.154>each <00:21:09.270>metric <00:21:09.387>is <00:21:09.504>going <00:21:09.620>to <00:21:09.737>range <00:21:09.854>on <00:21:09.970>a <00:21:10.087>scale <00:21:10.204>between <00:21:10.320>zero <00:21:10.437>and <00:21:10.554>one.

353
00:21:14.694 --> 00:21:18.443
<00:21:14.694>When <00:21:14.869>we <00:21:15.043>are <00:21:15.219>evaluating <00:21:15.394>using <00:21:15.569>the <00:21:15.744>test <00:21:15.918>set, <00:21:16.094>every

354
00:21:18.674 --> 00:21:22.354
<00:21:18.674>evaluated <00:21:18.907>row <00:21:19.140>will <00:21:19.374>have <00:21:19.607>four <00:21:19.840>components: <00:21:20.074>the

355
00:21:22.454 --> 00:21:26.114
<00:21:22.454>user <00:21:22.687>question, <00:21:22.920>synthetic <00:21:23.154>or <00:21:23.387>defined <00:21:23.620>by <00:21:23.854>you;

356
00:21:27.214 --> 00:21:31.204
<00:21:27.214>the <00:21:27.330>retrieved <00:21:27.447>context, <00:21:27.563>so <00:21:27.680>the <00:21:27.797>chunks <00:21:27.914>that <00:21:28.030>were <00:21:28.147>relevant; <00:21:28.264>the <00:21:28.380>response <00:21:28.497>that <00:21:28.614>your

357
00:21:31.254 --> 00:21:34.374
<00:21:31.254>RAG <00:21:31.394>system <00:21:31.533>generated; <00:21:31.674>and <00:21:31.813>the <00:21:31.954>reference, <00:21:32.093>which <00:21:32.234>is <00:21:32.373>the <00:21:32.514>gold <00:21:32.654>label.

358
00:21:34.894 --> 00:21:37.854
<00:21:34.894>Then <00:21:35.021>based <00:21:35.148>on <00:21:35.275>these, <00:21:35.403>for <00:21:35.530>each <00:21:35.657>row <00:21:35.784>we <00:21:35.912>can <00:21:36.039>define <00:21:36.166>some <00:21:36.294>metrics.

359
00:21:39.694 --> 00:21:42.554
<00:21:39.694>We <00:21:39.894>can <00:21:40.094>categorize <00:21:40.293>these <00:21:40.494>metrics <00:21:40.694>into <00:21:40.894>two <00:21:41.094>groups.

360
00:21:42.754 --> 00:21:46.714
<00:21:42.754>These <00:21:42.870>are <00:21:42.987>the <00:21:43.103>retrieval <00:21:43.220>metrics <00:21:43.337>that <00:21:43.454>are <00:21:43.570>related <00:21:43.687>to <00:21:43.803>the <00:21:43.920>retrieval <00:21:44.037>part <00:21:44.154>of

361
00:21:46.774 --> 00:21:50.554
<00:21:46.774>the <00:21:47.007>RAG <00:21:47.240>mechanism <00:21:47.474>and <00:21:47.707>the <00:21:47.940>generation <00:21:48.174>metrics

362
00:21:50.564 --> 00:21:54.434
<00:21:50.564>that <00:21:50.719>are <00:21:50.875>connected <00:21:51.030>to <00:21:51.186>how <00:21:51.341>generation <00:21:51.497>works <00:21:51.652>once <00:21:51.808>we <00:21:51.964>already

363
00:21:54.474 --> 00:21:56.004
<00:21:54.474>have <00:21:54.933>the <00:21:55.391>context <00:21:55.851>defined.

364
00:21:56.834 --> 00:22:00.554
<00:21:56.834>The <00:21:56.961>reason <00:21:57.088>why <00:21:57.215>it's <00:21:57.343>good <00:21:57.470>to <00:21:57.597>have <00:21:57.724>a <00:21:57.852>look <00:21:57.979>at <00:21:58.106>the <00:21:58.234>different

365
00:22:00.594 --> 00:22:04.474
<00:22:00.594>metrics, <00:22:00.749>because <00:22:00.905>we <00:22:01.060>can <00:22:01.216>localize <00:22:01.371>better <00:22:01.527>which <00:22:01.682>part <00:22:01.838>of <00:22:01.994>our

366
00:22:04.494 --> 00:22:08.264
<00:22:04.494>system <00:22:04.610>is <00:22:04.727>failing. <00:22:04.843>Quite <00:22:04.960>often, <00:22:05.077>you <00:22:05.194>might <00:22:05.310>only <00:22:05.427>have <00:22:05.543>an <00:22:05.660>issue <00:22:05.777>with <00:22:05.894>the

367
00:22:08.294 --> 00:22:11.984
<00:22:08.294>retrieval <00:22:08.494>part. <00:22:08.694>You <00:22:08.894>need <00:22:09.094>to <00:22:09.294>tweak <00:22:09.494>the <00:22:09.694>re-ranking

368
00:22:12.054 --> 00:22:15.894
<00:22:12.054>mechanism, <00:22:12.229>for <00:22:12.404>instance, <00:22:12.579>but <00:22:12.754>your <00:22:12.929>generation <00:22:13.104>is <00:22:13.279>working <00:22:13.454>properly.

369
00:22:17.604 --> 00:22:21.234
<00:22:17.604>Let's <00:22:17.697>have <00:22:17.790>a <00:22:17.884>look <00:22:17.977>at <00:22:18.070>first <00:22:18.164>the <00:22:18.257>retrieval <00:22:18.350>metrics, <00:22:18.444>and <00:22:18.537>then <00:22:18.630>we'll <00:22:18.724>move <00:22:18.817>over <00:22:18.910>to <00:22:19.004>the

370
00:22:21.374 --> 00:22:22.594
<00:22:21.374>generation <00:22:22.472>metrics.

371
00:22:24.194 --> 00:22:27.414
<00:22:24.194>So <00:22:24.369>in <00:22:24.543>the <00:22:24.719>retrieval, <00:22:24.894>essentially, <00:22:25.069>we <00:22:25.244>are <00:22:25.418>looking <00:22:25.594>at

372
00:22:29.054 --> 00:22:32.854
<00:22:29.054>whether <00:22:29.181>we <00:22:29.308>managed <00:22:29.435>to <00:22:29.563>gather <00:22:29.690>the <00:22:29.817>appropriate <00:22:29.944>chunks <00:22:30.072>to <00:22:30.199>answer <00:22:30.326>the <00:22:30.454>user

373
00:22:32.954 --> 00:22:33.374
query.

374
00:22:34.674 --> 00:22:38.414
<00:22:34.674>Context <00:22:34.849>precision <00:22:35.024>is <00:22:35.199>where <00:22:35.374>we <00:22:35.549>look <00:22:35.724>at <00:22:35.899>whether <00:22:36.074>the

375
00:22:38.434 --> 00:22:41.884
<00:22:38.434>relevant <00:22:38.589>chunks <00:22:38.745>that <00:22:38.900>are <00:22:39.056>most <00:22:39.211>helpful <00:22:39.367>to <00:22:39.522>answer <00:22:39.678>the <00:22:39.834>user's

376
00:22:41.954 --> 00:22:45.904
<00:22:41.954>query <00:22:42.070>are <00:22:42.187>ranked <00:22:42.303>at <00:22:42.420>the <00:22:42.537>top. <00:22:42.654>So <00:22:42.770>this <00:22:42.887>is <00:22:43.004>evaluating <00:22:43.120>your <00:22:43.237>ranking <00:22:43.354>strategy,

377
00:22:46.914 --> 00:22:50.693
<00:22:46.914>and <00:22:47.030>if <00:22:47.147>this <00:22:47.264>value <00:22:47.380>is <00:22:47.497>low, <00:22:47.614>it <00:22:47.730>might <00:22:47.847>mean <00:22:47.964>that <00:22:48.080>the <00:22:48.197>retriever <00:22:48.314>is

378
00:22:50.734 --> 00:22:53.894
<00:22:50.734>not <00:22:50.861>working <00:22:50.988>well <00:22:51.115>and <00:22:51.243>the <00:22:51.370>good <00:22:51.497>answers <00:22:51.624>are <00:22:51.752>lower <00:22:51.879>in <00:22:52.006>the <00:22:52.134>ranking.

379
00:22:56.034 --> 00:22:59.834
<00:22:56.034>On <00:22:56.161>the <00:22:56.288>other <00:22:56.415>hand, <00:22:56.543>you <00:22:56.670>have <00:22:56.797>context <00:22:56.924>recall, <00:22:57.052>which <00:22:57.179>evaluates <00:22:57.306>if <00:22:57.434>we

380
00:22:59.874 --> 00:23:03.614
<00:22:59.874>managed <00:23:00.074>to <00:23:00.274>get <00:23:00.474>all <00:23:00.674>of <00:23:00.874>the <00:23:01.074>relevant <00:23:01.274>chunks

381
00:23:04.414 --> 00:23:06.214
<00:23:04.414>when <00:23:04.647>we <00:23:04.880>are <00:23:05.114>answering <00:23:05.347>the <00:23:05.580>user's <00:23:05.814>query.

382
00:23:06.314 --> 00:23:08.594
<00:23:06.314>So <00:23:06.469>this <00:23:06.625>is <00:23:06.780>here <00:23:06.936>a <00:23:07.091>bit <00:23:07.247>more <00:23:07.402>about <00:23:07.558>the <00:23:07.714>coverage

383
00:23:09.574 --> 00:23:13.234
<00:23:09.574>and <00:23:09.807>if <00:23:10.040>this <00:23:10.274>value <00:23:10.507>is <00:23:10.740>low, <00:23:10.974>for

384
00:23:13.294 --> 00:23:16.894
<00:23:13.294>instance, <00:23:13.434>it <00:23:13.574>might <00:23:13.714>be <00:23:13.854>the <00:23:13.994>case <00:23:14.134>that <00:23:14.274>the <00:23:14.414>answer <00:23:14.554>needed <00:23:14.694>three,

385
00:23:17.774 --> 00:23:21.394
<00:23:17.774>but <00:23:17.914>your <00:23:18.053>system <00:23:18.194>only <00:23:18.333>managed <00:23:18.474>to <00:23:18.613>fetch <00:23:18.754>one <00:23:18.893>of <00:23:19.033>those <00:23:19.174>relevant

386
00:23:21.434 --> 00:23:25.204
<00:23:21.434>queries. <00:23:21.589>Please <00:23:21.745>note <00:23:21.900>that <00:23:22.056>this <00:23:22.211>is <00:23:22.367>a <00:23:22.522>classical <00:23:22.678>case <00:23:22.834>of

387
00:23:25.214 --> 00:23:29.154
<00:23:25.214>precision <00:23:25.341>versus <00:23:25.468>recall, <00:23:25.595>so <00:23:25.723>we <00:23:25.850>want <00:23:25.977>to <00:23:26.104>balance <00:23:26.232>this, <00:23:26.359>but <00:23:26.486>of <00:23:26.614>course

388
00:23:29.194 --> 00:23:32.954
<00:23:29.194>there's <00:23:29.349>a <00:23:29.505>trade-off. <00:23:29.660>If <00:23:29.816>you <00:23:29.971>are <00:23:30.127>increasing <00:23:30.282>one, <00:23:30.438>the <00:23:30.594>other

389
00:23:33.074 --> 00:23:36.494
<00:23:33.074>might <00:23:33.424>drop, <00:23:33.774>and <00:23:34.124>so <00:23:34.474>on.

390
00:23:37.794 --> 00:23:41.234
<00:23:37.794>Once <00:23:37.910>we <00:23:38.027>looked <00:23:38.144>at <00:23:38.260>those <00:23:38.377>metrics, <00:23:38.494>we <00:23:38.610>also <00:23:38.727>need <00:23:38.844>to <00:23:38.960>talk <00:23:39.077>about <00:23:39.194>the

391
00:23:41.274 --> 00:23:45.254
<00:23:41.274>generation <00:23:41.390>metrics. <00:23:41.507>So <00:23:41.623>this <00:23:41.740>is <00:23:41.857>again <00:23:41.974>focusing <00:23:42.090>on <00:23:42.207>what <00:23:42.323>does <00:23:42.440>the <00:23:42.557>LLM <00:23:42.674>do

392
00:23:45.334 --> 00:23:47.314
<00:23:45.334>once <00:23:45.614>the <00:23:45.894>context <00:23:46.174>is <00:23:46.454>already <00:23:46.734>available.

393
00:23:47.994 --> 00:23:51.654
<00:23:47.994>The <00:23:48.168>first <00:23:48.343>metric <00:23:48.519>here <00:23:48.694>is <00:23:48.869>faithfulness, <00:23:49.043>which <00:23:49.218>is <00:23:49.394>essentially

394
00:23:51.694 --> 00:23:54.694
<00:23:51.694>evaluating <00:23:51.927>does <00:23:52.160>your <00:23:52.394>LLM <00:23:52.627>hallucinate <00:23:52.860>or <00:23:53.094>not.

395
00:23:55.374 --> 00:23:58.914
<00:23:55.374>It <00:23:55.574>manages <00:23:55.774>to <00:23:55.974>catch <00:23:56.174>the <00:23:56.374>confidently <00:23:56.574>wrong <00:23:56.774>answers.

396
00:24:00.154 --> 00:24:04.074
<00:24:00.154>So <00:24:00.294>we <00:24:00.434>are <00:24:00.574>checking <00:24:00.714>is <00:24:00.854>everything <00:24:00.994>grounded <00:24:01.134>in <00:24:01.274>the <00:24:01.414>context <00:24:01.554>that

397
00:24:04.134 --> 00:24:04.894
<00:24:04.134>was <00:24:04.818>retrieved.

398
00:24:06.374 --> 00:24:08.834
<00:24:06.374>Another <00:24:06.654>metric <00:24:06.934>here <00:24:07.214>is <00:24:07.494>response <00:24:07.774>relevancy,

399
00:24:09.774 --> 00:24:13.694
<00:24:09.774>where <00:24:09.929>we <00:24:10.085>are <00:24:10.240>checking <00:24:10.396>the <00:24:10.551>RAG's <00:24:10.707>output <00:24:10.862>against <00:24:11.018>the <00:24:11.174>user

400
00:24:13.834 --> 00:24:17.314
<00:24:13.834>query <00:24:14.009>and <00:24:14.184>with <00:24:14.359>an <00:24:14.534>embedding <00:24:14.709>model <00:24:14.884>we <00:24:15.059>are <00:24:15.234>checking

401
00:24:17.334 --> 00:24:20.754
<00:24:17.334>whether <00:24:17.800>there <00:24:18.267>is <00:24:18.734>similarity

402
00:24:20.794 --> 00:24:24.494
<00:24:20.794>between <00:24:20.969>the <00:24:21.144>answer <00:24:21.319>and <00:24:21.494>the <00:24:21.669>original <00:24:21.844>question, <00:24:22.019>so <00:24:22.194>we

403
00:24:24.534 --> 00:24:28.154
<00:24:24.534>can <00:24:24.734>catch <00:24:24.934>cases <00:24:25.134>where <00:24:25.334>the <00:24:25.534>answers <00:24:25.734>are <00:24:25.934>padded

404
00:24:28.314 --> 00:24:29.654
<00:24:28.314>or <00:24:28.615>they <00:24:28.917>are <00:24:29.218>only <00:24:29.520>partial.

405
00:24:30.694 --> 00:24:34.534
<00:24:30.694>Note <00:24:30.821>that <00:24:30.948>for <00:24:31.075>the <00:24:31.203>metrics <00:24:31.330>up <00:24:31.457>until <00:24:31.584>now, <00:24:31.712>we <00:24:31.839>prefer <00:24:31.966>if <00:24:32.094>the

406
00:24:34.594 --> 00:24:38.546
<00:24:34.594>metric <00:24:34.734>is <00:24:34.874>high, <00:24:35.014>so <00:24:35.154>closer <00:24:35.294>to <00:24:35.434>one. <00:24:35.574>But <00:24:35.714>for <00:24:35.854>the <00:24:35.994>last

407
00:24:38.606 --> 00:24:42.526
<00:24:38.606>metric, <00:24:38.761>we <00:24:38.917>want <00:24:39.072>it <00:24:39.228>to <00:24:39.383>be <00:24:39.539>low. <00:24:39.694>Noise <00:24:39.850>sensitivity <00:24:40.006>tries

408
00:24:42.566 --> 00:24:46.416
<00:24:42.566>to <00:24:42.766>measure <00:24:42.966>whether <00:24:43.166>adding <00:24:43.366>some <00:24:43.566>junk <00:24:43.766>context <00:24:43.966>can

409
00:24:46.426 --> 00:24:50.286
<00:24:46.426>throw <00:24:46.581>the <00:24:46.737>LLM <00:24:46.892>off. <00:24:47.048>So <00:24:47.203>we <00:24:47.359>are <00:24:47.514>mixing <00:24:47.670>up <00:24:47.826>appropriate

410
00:24:50.346 --> 00:24:54.006
<00:24:50.346>chunks <00:24:50.486>also <00:24:50.626>with <00:24:50.766>irrelevant <00:24:50.906>ones <00:24:51.046>and <00:24:51.186>see <00:24:51.326>how <00:24:51.466>the <00:24:51.606>LLM <00:24:51.746>behaves

411
00:24:54.086 --> 00:24:55.466
here.

412
00:24:58.046 --> 00:25:01.716
<00:24:58.046>Once <00:24:58.201>you <00:24:58.357>have <00:24:58.512>your <00:24:58.668>evaluation <00:24:58.823>done <00:24:58.979>and <00:24:59.134>you <00:24:59.290>run <00:24:59.446>your

413
00:25:01.786 --> 00:25:04.786
<00:25:01.786>pipeline, <00:25:02.019>it's <00:25:02.252>time <00:25:02.486>to <00:25:02.719>evaluate <00:25:02.952>those <00:25:03.186>metrics.

414
00:25:05.526 --> 00:25:09.136
<00:25:05.526>And <00:25:05.626>this <00:25:05.726>is <00:25:05.826>where <00:25:05.926>for <00:25:06.026>each <00:25:06.126>of <00:25:06.226>the <00:25:06.326>rows, <00:25:06.426>each <00:25:06.526>of <00:25:06.626>the <00:25:06.726>queries, <00:25:06.826>you <00:25:06.926>will

415
00:25:09.246 --> 00:25:13.186
<00:25:09.246>have <00:25:09.421>the <00:25:09.596>values <00:25:09.771>available, <00:25:09.946>and <00:25:10.121>then <00:25:10.296>you <00:25:10.471>can <00:25:10.646>draw

416
00:25:13.226 --> 00:25:16.266
<00:25:13.226>the <00:25:13.506>mean <00:25:13.786>values <00:25:14.066>from <00:25:14.346>those <00:25:14.626>rows.

417
00:25:17.446 --> 00:25:21.126
<00:25:17.446>There's <00:25:17.586>a <00:25:17.726>couple <00:25:17.866>of <00:25:18.005>things <00:25:18.146>or <00:25:18.285>patterns <00:25:18.426>that <00:25:18.565>we <00:25:18.706>can <00:25:18.846>identify.

418
00:25:21.606 --> 00:25:25.525
<00:25:21.606>It <00:25:21.733>can <00:25:21.860>happen <00:25:21.987>that <00:25:22.115>you <00:25:22.242>have <00:25:22.369>high <00:25:22.496>faithfulness <00:25:22.624>and <00:25:22.751>relevancy, <00:25:22.878>but <00:25:23.006>for

419
00:25:25.566 --> 00:25:29.546
<00:25:25.566>some <00:25:25.693>reason <00:25:25.820>low <00:25:25.947>on <00:25:26.075>recall. <00:25:26.202>It <00:25:26.329>means <00:25:26.456>that <00:25:26.584>your <00:25:26.711>model <00:25:26.838>is <00:25:26.966>behaving

420
00:25:29.706 --> 00:25:33.486
<00:25:29.706>fine, <00:25:29.846>but <00:25:29.985>you <00:25:30.126>need <00:25:30.265>to <00:25:30.406>revisit <00:25:30.545>your <00:25:30.686>retrieval <00:25:30.825>strategy, <00:25:30.966>think <00:25:31.106>about

421
00:25:33.546 --> 00:25:37.436
<00:25:33.546>how <00:25:33.673>you're <00:25:33.800>doing <00:25:33.927>the <00:25:34.055>top <00:25:34.182>case <00:25:34.309>selection, <00:25:34.436>and <00:25:34.564>it's <00:25:34.691>probably <00:25:34.818>not <00:25:34.946>your

422
00:25:37.466 --> 00:25:39.686
<00:25:37.466>prompt <00:25:37.699>that's <00:25:37.932>the <00:25:38.166>source <00:25:38.399>of <00:25:38.632>the <00:25:38.866>issues.

423
00:25:40.826 --> 00:25:44.406
<00:25:40.826>On <00:25:40.953>the <00:25:41.080>other <00:25:41.207>hand, <00:25:41.335>there <00:25:41.462>can <00:25:41.589>be <00:25:41.716>the <00:25:41.844>issue <00:25:41.971>of <00:25:42.098>low <00:25:42.226>faithfulness.

424
00:25:44.906 --> 00:25:48.646
<00:25:44.906>This <00:25:45.013>refers <00:25:45.121>to <00:25:45.229>hallucination <00:25:45.336>again, <00:25:45.444>so <00:25:45.552>you <00:25:45.659>might <00:25:45.767>want <00:25:45.875>to <00:25:45.982>check <00:25:46.090>on <00:25:46.198>the <00:25:46.306>prompt

425
00:25:48.786 --> 00:25:51.106
<00:25:48.786>and <00:25:49.019>also <00:25:49.252>select <00:25:49.486>a <00:25:49.719>different <00:25:49.952>model <00:25:50.186>family.

426
00:25:53.166 --> 00:25:57.006
<00:25:53.166>And <00:25:53.306>it <00:25:53.446>can <00:25:53.586>also <00:25:53.726>happen <00:25:53.866>that <00:25:54.005>your <00:25:54.146>noise <00:25:54.285>sensitivity <00:25:54.426>is <00:25:54.566>higher.

427
00:25:57.066 --> 00:26:00.586
<00:25:57.066>It <00:25:57.241>means <00:25:57.416>that <00:25:57.591>chunks <00:25:57.766>are <00:25:57.941>very <00:25:58.116>much <00:25:58.291>derailing <00:25:58.466>the

428
00:26:00.626 --> 00:26:03.106
<00:26:00.626>answers <00:26:00.859>when <00:26:01.092>the <00:26:01.326>LLM <00:26:01.559>is <00:26:01.792>generating <00:26:02.026>them.

429
00:26:03.786 --> 00:26:07.566
<00:26:03.786>Here, <00:26:03.941>the <00:26:04.097>strategy <00:26:04.252>could <00:26:04.408>be <00:26:04.563>to <00:26:04.719>tighten <00:26:04.874>the <00:26:05.030>retrieval <00:26:05.186>and

430
00:26:07.686 --> 00:26:09.486
<00:26:07.686>refine <00:26:08.152>your <00:26:08.619>re-ranking <00:26:09.086>mechanism.

431
00:26:10.486 --> 00:26:14.426
<00:26:10.486>What <00:26:10.626>we <00:26:10.766>do <00:26:10.906>recommend <00:26:11.046>is <00:26:11.186>that <00:26:11.326>you <00:26:11.466>run <00:26:11.606>your <00:26:11.746>evaluation <00:26:11.886>pipeline

432
00:26:14.486 --> 00:26:18.426
<00:26:14.486>already <00:26:14.613>at <00:26:14.740>the <00:26:14.867>beginning <00:26:14.995>of <00:26:15.122>your <00:26:15.249>experiments <00:26:15.376>and <00:26:15.504>refining <00:26:15.631>your <00:26:15.758>RAG <00:26:15.886>system,

433
00:26:19.026 --> 00:26:21.966
<00:26:19.026>and <00:26:19.201>then <00:26:19.376>once <00:26:19.551>you <00:26:19.726>tweak <00:26:19.901>one <00:26:20.076>thing, <00:26:20.251>for <00:26:20.426>instance,

434
00:26:22.346 --> 00:26:25.886
<00:26:22.346>updating <00:26:22.546>the <00:26:22.746>system <00:26:22.946>prompt, <00:26:23.146>then <00:26:23.346>you <00:26:23.546>rerun <00:26:23.746>the

435
00:26:25.926 --> 00:26:29.786
<00:26:25.926>pipeline <00:26:26.126>again <00:26:26.326>and <00:26:26.525>see <00:26:26.726>how <00:26:26.926>the <00:26:27.126>values <00:26:27.326>change.

436
00:26:30.206 --> 00:26:33.986
<00:26:30.206>So <00:26:30.322>please <00:26:30.439>don't <00:26:30.555>try <00:26:30.672>to <00:26:30.789>change <00:26:30.906>multiple <00:26:31.022>things <00:26:31.139>at <00:26:31.255>once, <00:26:31.372>but <00:26:31.489>rather <00:26:31.606>go

437
00:26:34.046 --> 00:26:37.786
<00:26:34.046>step <00:26:34.173>by <00:26:34.300>step <00:26:34.427>and <00:26:34.555>then <00:26:34.682>have <00:26:34.809>an <00:26:34.936>understanding <00:26:35.064>of <00:26:35.191>what <00:26:35.318>has <00:26:35.446>changed.

438
00:26:40.886 --> 00:26:44.566
<00:26:40.886>And <00:26:41.026>now <00:26:41.166>it's <00:26:41.306>time <00:26:41.446>to <00:26:41.586>get <00:26:41.726>back <00:26:41.866>to <00:26:42.005>our <00:26:42.146>code <00:26:42.286>and

439
00:26:44.646 --> 00:26:47.306
<00:26:44.646>actually <00:26:44.846>do <00:26:45.046>that <00:26:45.245>to <00:26:45.446>evaluate <00:26:45.646>our <00:26:45.846>RAG <00:26:46.046>system.

440
00:26:49.186 --> 00:26:52.846
<00:26:49.186>What <00:26:49.302>we <00:26:49.419>do <00:26:49.535>next <00:26:49.652>is <00:26:49.769>that <00:26:49.886>we <00:26:50.002>run <00:26:50.119>the <00:26:50.235>RAG <00:26:50.352>pipeline <00:26:50.469>over <00:26:50.586>the

441
00:26:52.906 --> 00:26:55.846
<00:26:52.906>generated <00:26:53.106>test <00:26:53.306>set. <00:26:53.505>So <00:26:53.706>we <00:26:53.906>are <00:26:54.106>using <00:26:54.306>the

442
00:26:56.726 --> 00:27:00.266
<00:26:56.726>RAG <00:26:56.926>answer <00:26:57.126>function, <00:26:57.326>and <00:26:57.526>we <00:26:57.726>are <00:26:57.926>getting <00:26:58.126>the

443
00:27:01.506 --> 00:27:02.266
<00:27:01.506>top <00:27:02.190>four

444
00:27:03.706 --> 00:27:05.826
<00:27:03.706>chunks <00:27:04.055>for <00:27:04.406>the <00:27:04.755>final <00:27:05.106>answer.

445
00:27:07.126 --> 00:27:10.066
<00:27:07.126>And <00:27:07.301>then <00:27:07.476>we <00:27:07.651>are <00:27:07.826>getting <00:27:08.001>the <00:27:08.176>data <00:27:08.351>frame <00:27:08.526>out.

446
00:27:11.166 --> 00:27:15.146
<00:27:11.166>So <00:27:11.282>for <00:27:11.399>each <00:27:11.515>of <00:27:11.632>the <00:27:11.749>questions, <00:27:11.866>our <00:27:11.982>RAG <00:27:12.099>system <00:27:12.216>managed <00:27:12.332>to <00:27:12.449>generate <00:27:12.566>a

447
00:27:15.206 --> 00:27:16.876
<00:27:15.206>response <00:27:15.485>that <00:27:15.765>we <00:27:16.045>see <00:27:16.325>right <00:27:16.606>here.

448
00:27:18.306 --> 00:27:22.106
<00:27:18.306>And <00:27:18.446>then <00:27:18.586>it's <00:27:18.726>time <00:27:18.866>to <00:27:19.006>go <00:27:19.146>ahead <00:27:19.286>and <00:27:19.426>score <00:27:19.566>with <00:27:19.706>the

449
00:27:22.166 --> 00:27:23.646
<00:27:22.166>five <00:27:22.831>Ragas <00:27:23.498>metrics.

450
00:27:24.766 --> 00:27:28.646
<00:27:24.766>So <00:27:24.921>here <00:27:25.077>we're <00:27:25.232>using <00:27:25.388>ideally <00:27:25.543>a <00:27:25.699>separate <00:27:25.854>large <00:27:26.010>language <00:27:26.166>model

451
00:27:28.686 --> 00:27:29.826
<00:27:28.686>to <00:27:28.942>act <00:27:29.199>as <00:27:29.455>a <00:27:29.712>judge,

452
00:27:31.006 --> 00:27:34.206
<00:27:31.006>and <00:27:31.206>we <00:27:31.406>are <00:27:31.606>using <00:27:31.806>the <00:27:32.006>previously <00:27:32.206>defined <00:27:32.406>metrics.

453
00:27:35.126 --> 00:27:35.546
One

454
00:27:36.386 --> 00:27:39.366
<00:27:36.386>thing <00:27:36.541>I <00:27:36.697>would <00:27:36.852>like <00:27:37.008>to <00:27:37.163>raise <00:27:37.319>awareness <00:27:37.474>here <00:27:37.630>is <00:27:37.786>that

455
00:27:40.646 --> 00:27:43.246
<00:27:40.646>in <00:27:40.846>the <00:27:41.046>presentation, <00:27:41.245>I <00:27:41.446>was <00:27:41.646>talking <00:27:41.846>about <00:27:42.046>response

456
00:27:43.306 --> 00:27:47.246
<00:27:43.306>relevancy. <00:27:43.446>In <00:27:43.586>some <00:27:43.726>versions <00:27:43.866>of <00:27:44.006>the <00:27:44.146>Ragas <00:27:44.286>library, <00:27:44.426>this <00:27:44.566>is <00:27:44.706>called

457
00:27:47.326 --> 00:27:50.706
<00:27:47.326>answer <00:27:47.501>relevancy, <00:27:47.676>but <00:27:47.851>they <00:27:48.026>refer <00:27:48.201>to <00:27:48.376>the <00:27:48.551>same <00:27:48.726>thing.

458
00:27:53.046 --> 00:27:56.886
<00:27:53.046>So <00:27:53.173>what <00:27:53.300>we <00:27:53.427>do <00:27:53.555>next <00:27:53.682>is <00:27:53.809>we <00:27:53.936>are <00:27:54.064>setting <00:27:54.191>up <00:27:54.318>the <00:27:54.446>client

459
00:27:56.966 --> 00:27:59.486
<00:27:56.966>again, <00:27:57.166>and <00:27:57.366>we <00:27:57.565>are <00:27:57.765>setting <00:27:57.966>up <00:27:58.166>the <00:27:58.366>judge.

460
00:28:00.486 --> 00:28:04.446
<00:28:00.486>For <00:28:00.602>the <00:28:00.719>simplicity <00:28:00.836>of <00:28:00.952>the <00:28:01.069>experiment, <00:28:01.186>I'm <00:28:01.302>just <00:28:01.419>going <00:28:01.536>again <00:28:01.652>with <00:28:01.769>GPT-4o <00:28:01.886>mini,

461
00:28:04.526 --> 00:28:06.166
<00:28:04.526>but <00:28:04.806>again, <00:28:05.086>you <00:28:05.366>might <00:28:05.646>want <00:28:05.926>to

462
00:28:07.026 --> 00:28:10.186
<00:28:07.026>update <00:28:07.153>to <00:28:07.280>a <00:28:07.407>different <00:28:07.535>model <00:28:07.662>or <00:28:07.789>a <00:28:07.916>higher <00:28:08.044>class <00:28:08.171>model <00:28:08.298>for <00:28:08.426>this

463
00:28:10.226 --> 00:28:13.956
<00:28:10.226>experiment. <00:28:10.401>And <00:28:10.576>we <00:28:10.751>are <00:28:10.926>also <00:28:11.101>using <00:28:11.276>embeddings <00:28:11.451>for <00:28:11.626>the

464
00:28:14.126 --> 00:28:17.726
<00:28:14.126>response <00:28:14.281>relevancy <00:28:14.437>because <00:28:14.592>that's <00:28:14.748>the <00:28:14.903>one <00:28:15.059>where <00:28:15.214>we <00:28:15.370>need <00:28:15.526>to

465
00:28:17.786 --> 00:28:19.426
<00:28:17.786>rely <00:28:18.252>on <00:28:18.719>consent <00:28:19.186>similarity.

466
00:28:20.826 --> 00:28:22.906
<00:28:20.826>We <00:28:21.059>have <00:28:21.292>defined <00:28:21.526>all <00:28:21.759>of <00:28:21.992>the <00:28:22.226>metrics,

467
00:28:23.806 --> 00:28:27.106
<00:28:23.806>and <00:28:23.946>then <00:28:24.086>we <00:28:24.226>are <00:28:24.366>waiting <00:28:24.506>for <00:28:24.646>the <00:28:24.786>LLM <00:28:24.926>as <00:28:25.066>judge <00:28:25.206>to

468
00:28:27.186 --> 00:28:31.086
<00:28:27.186>evaluate <00:28:27.326>each <00:28:27.466>of <00:28:27.606>the <00:28:27.745>metrics <00:28:27.886>for <00:28:28.025>each <00:28:28.166>one <00:28:28.305>of <00:28:28.446>the <00:28:28.586>rows

469
00:28:31.126 --> 00:28:32.646
<00:28:31.126>in <00:28:31.468>our <00:28:31.810>synthetic <00:28:32.152>test <00:28:32.494>set.

470
00:28:33.546 --> 00:28:35.536
<00:28:33.546>Note <00:28:33.779>that <00:28:34.012>this <00:28:34.246>can <00:28:34.479>take <00:28:34.712>some <00:28:34.946>time.

471
00:28:36.766 --> 00:28:40.526
<00:28:36.766>And <00:28:36.893>once <00:28:37.020>that <00:28:37.147>happened, <00:28:37.275>the <00:28:37.402>scoring <00:28:37.529>is <00:28:37.656>done <00:28:37.784>on <00:28:37.911>each <00:28:38.038>of <00:28:38.166>the

472
00:28:40.586 --> 00:28:44.366
<00:28:40.586>rows. <00:28:40.786>We <00:28:40.986>can <00:28:41.186>just <00:28:41.386>save <00:28:41.586>the <00:28:41.786>results <00:28:41.986>into

473
00:28:44.506 --> 00:28:48.426
<00:28:44.506>a <00:28:44.593>CSV <00:28:44.681>file, <00:28:44.768>have <00:28:44.856>a <00:28:44.943>look <00:28:45.031>at <00:28:45.118>it <00:28:45.206>later, <00:28:45.293>or <00:28:45.381>just <00:28:45.468>have <00:28:45.556>it <00:28:45.643>in <00:28:45.731>the <00:28:45.818>data <00:28:45.906>frame.

474
00:28:49.246 --> 00:28:50.886
<00:28:49.246>And <00:28:49.446>let's <00:28:49.646>have <00:28:49.846>a <00:28:50.046>look <00:28:50.246>at <00:28:50.446>what <00:28:50.646>happened.

475
00:28:51.726 --> 00:28:54.045
<00:28:51.726>We <00:28:51.926>had <00:28:52.126>a <00:28:52.326>look <00:28:52.526>at <00:28:52.726>the <00:28:52.926>original <00:28:53.126>question,

476
00:28:55.306 --> 00:28:58.906
<00:28:55.306>and <00:28:55.422>of <00:28:55.539>course, <00:28:55.656>in <00:28:55.772>the <00:28:55.889>data, <00:28:56.006>we <00:28:56.122>also <00:28:56.239>see <00:28:56.356>the <00:28:56.472>goal <00:28:56.589>label, <00:28:56.706>the

477
00:28:58.946 --> 00:29:02.546
<00:28:58.946>generated <00:28:59.120>label, <00:28:59.295>the <00:28:59.471>relevant <00:28:59.646>chunks, <00:28:59.821>and <00:28:59.995>then <00:29:00.170>all <00:29:00.346>of

478
00:29:02.606 --> 00:29:04.856
<00:29:02.606>the <00:29:02.956>metrics <00:29:03.306>here <00:29:03.656>are <00:29:04.006>visible.

479
00:29:05.886 --> 00:29:09.716
<00:29:05.886>And <00:29:06.002>what <00:29:06.119>we <00:29:06.235>can <00:29:06.352>do <00:29:06.469>next <00:29:06.586>is <00:29:06.702>either <00:29:06.819>go <00:29:06.936>ahead <00:29:07.052>and <00:29:07.169>look <00:29:07.286>into

480
00:29:09.746 --> 00:29:11.346
<00:29:09.746>each <00:29:09.979>of <00:29:10.212>the <00:29:10.446>questions <00:29:10.679>one <00:29:10.912>by <00:29:11.146>one,

481
00:29:13.046 --> 00:29:17.026
<00:29:13.046>or <00:29:13.162>we <00:29:13.279>can <00:29:13.396>just <00:29:13.512>take <00:29:13.629>the <00:29:13.746>mean <00:29:13.862>values <00:29:13.979>here <00:29:14.096>and <00:29:14.212>then <00:29:14.329>have <00:29:14.446>a

482
00:29:17.066 --> 00:29:20.886
<00:29:17.066>deeper <00:29:17.193>look <00:29:17.320>at <00:29:17.447>each <00:29:17.575>of <00:29:17.702>them, <00:29:17.829>start <00:29:17.956>thinking <00:29:18.084>about <00:29:18.211>what <00:29:18.338>can <00:29:18.466>we

483
00:29:20.946 --> 00:29:24.266
<00:29:20.946>do <00:29:21.073>to <00:29:21.200>tweak <00:29:21.327>the <00:29:21.455>system <00:29:21.582>to <00:29:21.709>adjust <00:29:21.836>the <00:29:21.964>values <00:29:22.091>so <00:29:22.218>that <00:29:22.346>they

484
00:29:24.326 --> 00:29:27.966
<00:29:24.326>are <00:29:24.466>higher <00:29:24.606>or <00:29:24.746>lower <00:29:24.886>respective <00:29:25.026>of <00:29:25.166>what <00:29:25.306>we're <00:29:25.446>trying <00:29:25.586>to <00:29:25.726>achieve.

485
00:29:33.706 --> 00:29:37.346
<00:29:33.706>What <00:29:33.822>I <00:29:33.939>would <00:29:34.055>also <00:29:34.172>like <00:29:34.289>to <00:29:34.406>mention <00:29:34.522>at <00:29:34.639>this <00:29:34.755>point <00:29:34.872>is <00:29:34.989>that <00:29:35.106>generating

486
00:29:37.426 --> 00:29:40.806
<00:29:37.426>synthetic <00:29:37.581>data <00:29:37.737>is <00:29:37.892>not <00:29:38.048>the <00:29:38.203>final <00:29:38.359>step <00:29:38.514>of <00:29:38.670>a <00:29:38.826>proper

487
00:29:40.846 --> 00:29:44.266
<00:29:40.846>evaluation <00:29:40.986>pipeline. <00:29:41.126>It <00:29:41.266>is <00:29:41.406>only <00:29:41.546>a <00:29:41.686>bootstrap <00:29:41.826>but <00:29:41.966>not <00:29:42.106>a <00:29:42.246>verdict.

488
00:29:44.586 --> 00:29:45.706
<00:29:44.586>It's <00:29:44.838>not <00:29:45.090>the <00:29:45.341>final <00:29:45.593>step.

489
00:29:46.966 --> 00:29:50.846
<00:29:46.966>You <00:29:47.073>need <00:29:47.181>to <00:29:47.289>know <00:29:47.396>that <00:29:47.504>wherein <00:29:47.612>we <00:29:47.719>are <00:29:47.827>using <00:29:47.935>LLM <00:29:48.042>as <00:29:48.150>judge, <00:29:48.258>we <00:29:48.366>are

490
00:29:50.946 --> 00:29:53.306
<00:29:50.946>measuring <00:29:51.646>synthetic <00:29:52.346>data

491
00:29:54.106 --> 00:29:57.506
<00:29:54.106>against <00:29:54.281>a <00:29:54.456>RAG <00:29:54.631>system <00:29:54.806>that <00:29:54.981>again <00:29:55.156>uses <00:29:55.331>an <00:29:55.506>LLM.

492
00:29:57.646 --> 00:29:59.806
<00:29:57.646>So <00:29:57.821>of <00:29:57.995>course <00:29:58.171>it <00:29:58.346>is <00:29:58.521>going <00:29:58.696>to <00:29:58.870>have <00:29:59.046>biases.

493
00:30:00.366 --> 00:30:04.306
<00:30:00.366>There <00:30:00.473>might <00:30:00.581>be <00:30:00.689>blind <00:30:00.796>spots <00:30:00.904>in <00:30:01.012>your <00:30:01.119>system, <00:30:01.227>especially <00:30:01.335>if <00:30:01.442>they <00:30:01.550>are <00:30:01.658>from <00:30:01.766>the

494
00:30:04.426 --> 00:30:05.406
<00:30:04.426>same <00:30:04.867>model <00:30:05.308>family.

495
00:30:07.106 --> 00:30:09.066
<00:30:07.106>Generated <00:30:07.456>questions <00:30:07.806>might <00:30:08.156>be <00:30:08.506>shallow.

496
00:30:09.966 --> 00:30:13.706
<00:30:09.966>Although <00:30:10.106>we <00:30:10.245>define <00:30:10.386>the <00:30:10.525>persona <00:30:10.666>and <00:30:10.805>we <00:30:10.946>try <00:30:11.085>to <00:30:11.226>go <00:30:11.366>with

497
00:30:13.726 --> 00:30:17.266
<00:30:13.726>different <00:30:13.901>types <00:30:14.076>of <00:30:14.251>questions, <00:30:14.426>abstracts, <00:30:14.601>specific, <00:30:14.776>and <00:30:14.951>of <00:30:15.126>course

498
00:30:17.326 --> 00:30:21.306
<00:30:17.326>single-hop <00:30:17.501>and <00:30:17.676>multi-hop, <00:30:17.851>it <00:30:18.026>can <00:30:18.201>still <00:30:18.376>happen <00:30:18.551>that <00:30:18.726>you

499
00:30:21.326 --> 00:30:22.986
<00:30:21.326>leave <00:30:21.559>the <00:30:21.792>answer <00:30:22.026>straight <00:30:22.259>from <00:30:22.492>the <00:30:22.726>chunk.

500
00:30:23.326 --> 00:30:24.346
<00:30:23.326>That <00:30:23.632>is <00:30:23.938>part <00:30:24.244>of

501
00:30:25.166 --> 00:30:25.426
this.

502
00:30:26.606 --> 00:30:29.926
<00:30:26.606>Of <00:30:26.761>course, <00:30:26.917>the <00:30:27.072>gold <00:30:27.228>labels <00:30:27.383>might <00:30:27.539>not <00:30:27.694>be <00:30:27.850>properly <00:30:28.006>correct.

503
00:30:30.046 --> 00:30:33.826
<00:30:30.046>So <00:30:30.173>I <00:30:30.300>do <00:30:30.427>recommend <00:30:30.555>that <00:30:30.682>you <00:30:30.809>have <00:30:30.936>a <00:30:31.064>human <00:30:31.191>that <00:30:31.318>is <00:30:31.446>verifying

504
00:30:33.886 --> 00:30:37.706
<00:30:33.886>that <00:30:34.061>synthetic <00:30:34.235>test <00:30:34.411>set. <00:30:34.586>They <00:30:34.761>might <00:30:34.936>be <00:30:35.110>tweaking <00:30:35.286>the

505
00:30:37.766 --> 00:30:40.266
<00:30:37.766>reference <00:30:37.966>a <00:30:38.166>little <00:30:38.366>bit, <00:30:38.566>the <00:30:38.766>gold <00:30:38.966>labels <00:30:39.166>or

506
00:30:41.270 --> 00:30:44.770
<00:30:41.270>You <00:30:41.386>might <00:30:41.503>want <00:30:41.620>to <00:30:41.736>have <00:30:41.853>an <00:30:41.970>actual <00:30:42.086>human <00:30:42.203>that <00:30:42.320>is <00:30:42.436>writing <00:30:42.553>these <00:30:42.670>queries

507
00:30:45.550 --> 00:30:49.270
<00:30:45.550>and <00:30:45.725>answers <00:30:45.899>for <00:30:46.075>you. <00:30:46.250>And <00:30:46.425>these <00:30:46.600>metrics <00:30:46.774>are <00:30:46.950>only

508
00:30:49.310 --> 00:30:53.070
<00:30:49.310>proxies. <00:30:49.397>Of <00:30:49.485>course, <00:30:49.572>when <00:30:49.659>you <00:30:49.747>are <00:30:49.835>setting <00:30:49.922>up <00:30:50.010>a <00:30:50.097>RAG <00:30:50.185>system, <00:30:50.272>there <00:30:50.360>is <00:30:50.447>a <00:30:50.534>lot <00:30:50.622>more <00:30:50.710>going

509
00:30:53.170 --> 00:30:56.010
<00:30:53.170>into <00:30:53.310>it. <00:30:53.450>You <00:30:53.590>need <00:30:53.730>to <00:30:53.870>think <00:30:54.010>about <00:30:54.150>the <00:30:54.290>whole <00:30:54.430>user <00:30:54.570>experience.

510
00:30:56.750 --> 00:31:00.170
<00:30:56.750>Of <00:30:56.877>course, <00:30:57.004>UX <00:30:57.131>is <00:30:57.259>part <00:30:57.386>of <00:30:57.513>that, <00:30:57.640>so <00:30:57.768>these <00:30:57.895>metrics <00:30:58.022>do <00:30:58.150>not

511
00:31:00.810 --> 00:31:03.110
<00:31:00.810>always <00:31:01.159>mean <00:31:01.510>good <00:31:01.860>user <00:31:02.210>satisfaction.

512
00:31:03.750 --> 00:31:07.570
<00:31:03.750>There <00:31:03.843>is <00:31:03.936>a <00:31:04.030>lot <00:31:04.123>more <00:31:04.216>that <00:31:04.310>can <00:31:04.403>go <00:31:04.496>into <00:31:04.590>building <00:31:04.683>an <00:31:04.776>application, <00:31:04.870>and <00:31:04.963>of <00:31:05.056>course, <00:31:05.150>this

513
00:31:07.630 --> 00:31:11.540
<00:31:07.630>is <00:31:07.830>just <00:31:08.030>one <00:31:08.230>part <00:31:08.430>of <00:31:08.630>that <00:31:08.830>whole <00:31:09.030>thought

514
00:31:11.590 --> 00:31:13.230
<00:31:11.590>process <00:31:11.870>that <00:31:12.149>must <00:31:12.429>go <00:31:12.709>into <00:31:12.990>it.

515
00:31:16.570 --> 00:31:20.520
<00:31:16.570>And <00:31:16.710>it's <00:31:16.850>also <00:31:16.990>important <00:31:17.129>to <00:31:17.270>emphasize <00:31:17.409>that <00:31:17.550>we <00:31:17.689>need <00:31:17.830>to <00:31:17.970>keep

516
00:31:20.570 --> 00:31:23.810
<00:31:20.570>this <00:31:20.710>as <00:31:20.850>honest <00:31:20.990>as <00:31:21.129>possible. <00:31:21.270>So <00:31:21.409>I <00:31:21.550>do <00:31:21.689>recommend <00:31:21.830>to <00:31:21.970>involve

517
00:31:23.890 --> 00:31:26.990
<00:31:23.890>humans <00:31:24.065>in <00:31:24.240>the <00:31:24.415>evaluation <00:31:24.590>process <00:31:24.765>and <00:31:24.940>in <00:31:25.115>the <00:31:25.290>data

518
00:31:27.990 --> 00:31:31.860
<00:31:27.990>generation <00:31:28.223>process <00:31:28.456>as <00:31:28.690>well. <00:31:28.923>You <00:31:29.156>can <00:31:29.390>always

519
00:31:32.210 --> 00:31:35.890
<00:31:32.210>decide <00:31:32.350>to <00:31:32.490>use <00:31:32.630>a <00:31:32.770>small <00:31:32.910>set <00:31:33.050>of <00:31:33.190>human <00:31:33.330>labels <00:31:33.470>over <00:31:33.610>the

520
00:31:35.950 --> 00:31:39.910
<00:31:35.950>synthetic <00:31:36.066>test <00:31:36.183>data, <00:31:36.300>or <00:31:36.416>you <00:31:36.533>might <00:31:36.650>want <00:31:36.766>to <00:31:36.883>have <00:31:37.000>a <00:31:37.116>bit <00:31:37.233>of <00:31:37.350>both.

521
00:31:40.790 --> 00:31:44.250
<00:31:40.790>I <00:31:40.930>do <00:31:41.070>recommend <00:31:41.210>using <00:31:41.350>a <00:31:41.490>stronger <00:31:41.629>model <00:31:41.770>to <00:31:41.909>be <00:31:42.050>your <00:31:42.190>judge,

522
00:31:44.690 --> 00:31:48.410
<00:31:44.690>and <00:31:44.845>it's <00:31:45.001>a <00:31:45.156>good <00:31:45.312>idea <00:31:45.467>to <00:31:45.623>version <00:31:45.778>and <00:31:45.934>always <00:31:46.090>refine

523
00:31:48.470 --> 00:31:52.370
<00:31:48.470>your <00:31:48.625>generated <00:31:48.781>test <00:31:48.936>set. <00:31:49.092>You <00:31:49.247>might <00:31:49.403>want <00:31:49.558>to <00:31:49.714>add <00:31:49.870>some

524
00:31:52.390 --> 00:31:56.140
<00:31:52.390>more <00:31:52.530>questions <00:31:52.670>or <00:31:52.810>remove <00:31:52.950>them <00:31:53.090>if <00:31:53.230>you <00:31:53.370>notice <00:31:53.510>that <00:31:53.650>the <00:31:53.790>metrics

525
00:31:56.210 --> 00:31:59.660
<00:31:56.210>are <00:31:56.410>off, <00:31:56.610>or <00:31:56.810>your <00:31:57.010>subject <00:31:57.210>matter <00:31:57.410>expert <00:31:57.610>might

526
00:31:59.730 --> 00:32:02.210
<00:31:59.730>overwrite <00:32:00.080>what <00:32:00.430>the <00:32:00.780>LLM <00:32:01.130>generated.

527
00:32:03.370 --> 00:32:07.340
<00:32:03.370>And <00:32:03.463>the <00:32:03.556>nice <00:32:03.649>thing <00:32:03.743>is <00:32:03.836>that <00:32:03.929>once <00:32:04.023>you <00:32:04.116>have <00:32:04.209>your <00:32:04.303>system <00:32:04.396>in <00:32:04.489>production, <00:32:04.583>you <00:32:04.676>can <00:32:04.770>always

528
00:32:07.390 --> 00:32:10.410
<00:32:07.390>mix <00:32:07.545>in <00:32:07.701>some <00:32:07.856>real <00:32:08.012>production <00:32:08.167>queries <00:32:08.323>into <00:32:08.478>the <00:32:08.634>test <00:32:08.790>set.

529
00:32:11.190 --> 00:32:15.150
<00:32:11.190>And <00:32:11.290>of <00:32:11.390>course, <00:32:11.490>keep <00:32:11.590>the <00:32:11.690>test <00:32:11.790>set <00:32:11.890>versions, <00:32:11.990>keep <00:32:12.090>the <00:32:12.190>prompts <00:32:12.290>versions, <00:32:12.390>so <00:32:12.490>that <00:32:12.590>you

530
00:32:15.190 --> 00:32:17.630
<00:32:15.190>can <00:32:15.423>always <00:32:15.656>roll <00:32:15.890>back <00:32:16.123>to <00:32:16.356>previous <00:32:16.590>versions.

531
00:32:19.250 --> 00:32:22.050
<00:32:19.250>To <00:32:19.483>talk <00:32:19.716>about <00:32:19.950>the <00:32:20.183>maturity <00:32:20.416>timeline <00:32:20.650>of

532
00:32:23.570 --> 00:32:27.450
<00:32:23.570>how <00:32:23.669>this <00:32:23.770>whole <00:32:23.870>thing <00:32:23.970>can <00:32:24.070>come <00:32:24.169>together, <00:32:24.270>I <00:32:24.370>think <00:32:24.470>it's <00:32:24.570>a <00:32:24.669>good <00:32:24.770>idea <00:32:24.870>to <00:32:24.970>already

533
00:32:27.530 --> 00:32:31.090
<00:32:27.530>start <00:32:27.657>with <00:32:27.784>a <00:32:27.911>synthetic <00:32:28.039>test <00:32:28.166>set. <00:32:28.293>Even <00:32:28.420>when <00:32:28.548>you're <00:32:28.675>building <00:32:28.802>your <00:32:28.930>RAG

534
00:32:31.150 --> 00:32:35.080
<00:32:31.150>system, <00:32:31.325>evaluate <00:32:31.500>from <00:32:31.675>the <00:32:31.850>beginning, <00:32:32.025>even <00:32:32.200>if <00:32:32.375>you <00:32:32.550>don't

535
00:32:35.130 --> 00:32:39.030
<00:32:35.130>have <00:32:35.270>user <00:32:35.410>data <00:32:35.550>yet. <00:32:35.690>And <00:32:35.830>then <00:32:35.970>once <00:32:36.110>you <00:32:36.250>have <00:32:36.390>data, <00:32:36.530>you

536
00:32:39.150 --> 00:32:42.790
<00:32:39.150>start <00:32:39.266>to <00:32:39.383>collect <00:32:39.500>some <00:32:39.616>data <00:32:39.733>with <00:32:39.850>the <00:32:39.966>approval <00:32:40.083>of <00:32:40.200>the <00:32:40.316>users, <00:32:40.433>of <00:32:40.550>course.

537
00:32:43.270 --> 00:32:47.010
<00:32:43.270>Then <00:32:43.445>these <00:32:43.620>can <00:32:43.795>be <00:32:43.970>added <00:32:44.145>to <00:32:44.320>the <00:32:44.495>test <00:32:44.670>set,

538
00:32:47.190 --> 00:32:50.650
<00:32:47.190>especially <00:32:47.390>if <00:32:47.590>your <00:32:47.790>RAG <00:32:47.990>system <00:32:48.190>failed <00:32:48.390>on <00:32:48.590>them.

539
00:32:51.670 --> 00:32:55.560
<00:32:51.670>Then <00:32:51.797>it's <00:32:51.924>a <00:32:52.051>good <00:32:52.179>idea <00:32:52.306>to <00:32:52.433>check <00:32:52.560>why <00:32:52.688>that <00:32:52.815>failed, <00:32:52.942>annotate <00:32:53.070>it,

540
00:32:55.650 --> 00:32:57.530
<00:32:55.650>and <00:32:55.825>then <00:32:56.000>put <00:32:56.175>it <00:32:56.350>back <00:32:56.525>to <00:32:56.700>the <00:32:56.875>test <00:32:57.050>set.

541
00:33:01.010 --> 00:33:03.270
<00:33:01.010>To <00:33:01.243>recap <00:33:01.476>of <00:33:01.710>what <00:33:01.943>we've <00:33:02.176>talked <00:33:02.410>about,

542
00:33:04.570 --> 00:33:08.450
<00:33:04.570>when <00:33:04.710>you <00:33:04.850>want <00:33:04.990>to <00:33:05.129>set <00:33:05.270>up <00:33:05.409>RAGAS <00:33:05.550>for <00:33:05.689>your <00:33:05.830>eval <00:33:05.970>pipelines,

543
00:33:08.930 --> 00:33:11.210
<00:33:08.930>you <00:33:09.105>only <00:33:09.280>need <00:33:09.455>to <00:33:09.630>point <00:33:09.805>it <00:33:09.980>at <00:33:10.155>your <00:33:10.330>documents,

544
00:33:12.030 --> 00:33:15.670
<00:33:12.030>generate <00:33:12.157>a <00:33:12.284>test <00:33:12.411>set, <00:33:12.539>and <00:33:12.666>then <00:33:12.793>run <00:33:12.920>the <00:33:13.048>evaluation <00:33:13.175>pipeline <00:33:13.302>on <00:33:13.430>it.

545
00:33:16.410 --> 00:33:17.010
<00:33:16.410>After <00:33:16.950>that,

546
00:33:18.250 --> 00:33:20.990
<00:33:18.250>you <00:33:18.450>can <00:33:18.650>score <00:33:18.850>the <00:33:19.050>retrieval <00:33:19.250>and <00:33:19.450>the <00:33:19.650>generation.

547
00:33:21.170 --> 00:33:24.670
<00:33:21.170>You <00:33:21.286>will <00:33:21.403>have <00:33:21.520>different <00:33:21.636>metrics, <00:33:21.753>the <00:33:21.870>five <00:33:21.986>metrics <00:33:22.103>we <00:33:22.220>have <00:33:22.336>talked <00:33:22.453>about <00:33:22.570>today.

548
00:33:25.710 --> 00:33:29.590
<00:33:25.710>Here <00:33:25.826>comes <00:33:25.943>the <00:33:26.060>essential <00:33:26.176>part, <00:33:26.293>the <00:33:26.410>human <00:33:26.526>part, <00:33:26.643>that <00:33:26.760>is <00:33:26.876>looking <00:33:26.993>at <00:33:27.110>and

549
00:33:29.630 --> 00:33:32.970
<00:33:29.630>diagnosing <00:33:29.770>these <00:33:29.910>metrics <00:33:30.050>and <00:33:30.190>try <00:33:30.330>to <00:33:30.470>understand <00:33:30.610>what <00:33:30.750>they <00:33:30.890>mean <00:33:31.030>for

550
00:33:33.070 --> 00:33:36.730
<00:33:33.070>you. <00:33:33.210>Then <00:33:33.350>you <00:33:33.490>can <00:33:33.629>go <00:33:33.770>back <00:33:33.909>to <00:33:34.050>tweak <00:33:34.189>your <00:33:34.330>system, <00:33:34.470>fix

551
00:33:36.790 --> 00:33:38.570
<00:33:36.790>something, <00:33:37.070>and <00:33:37.350>then <00:33:37.629>rerun <00:33:37.909>it <00:33:38.190>again.

552
00:33:40.130 --> 00:33:43.930
<00:33:40.130>I <00:33:40.305>recommend <00:33:40.480>to <00:33:40.655>think <00:33:40.830>about <00:33:41.005>evaluation <00:33:41.180>and <00:33:41.355>observability <00:33:41.530>earlier

553
00:33:44.010 --> 00:33:47.750
<00:33:44.010>than <00:33:44.110>you <00:33:44.210>roll <00:33:44.310>it <00:33:44.410>out <00:33:44.510>to <00:33:44.610>your <00:33:44.710>users. <00:33:44.810>You <00:33:44.910>save <00:33:45.010>yourself <00:33:45.110>a <00:33:45.210>lot <00:33:45.310>of <00:33:45.410>headache

554
00:33:47.890 --> 00:33:48.350
<00:33:47.890>if <00:33:48.303>you

555
00:33:49.310 --> 00:33:53.070
<00:33:49.310>do <00:33:49.437>this <00:33:49.564>from <00:33:49.691>day <00:33:49.819>one. <00:33:49.946>And <00:33:50.073>don't <00:33:50.200>shy <00:33:50.328>away <00:33:50.455>from <00:33:50.582>using <00:33:50.710>synthetic

556
00:33:53.150 --> 00:33:57.090
<00:33:53.150>data. <00:33:53.305>It <00:33:53.461>can <00:33:53.616>be <00:33:53.772>very <00:33:53.927>useful <00:33:54.083>for <00:33:54.238>setting <00:33:54.394>up <00:33:54.550>evaluations.

557
00:33:58.330 --> 00:34:01.970
<00:33:58.330>And <00:33:58.417>to <00:33:58.504>wrap <00:33:58.592>this <00:33:58.679>all <00:33:58.767>up, <00:33:58.855>I <00:33:58.942>would <00:33:59.030>like <00:33:59.117>to <00:33:59.205>thank <00:33:59.292>you <00:33:59.379>for <00:33:59.467>your <00:33:59.554>time <00:33:59.642>and <00:33:59.730>attention

558
00:34:02.230 --> 00:34:03.050
<00:34:02.230>for <00:34:02.476>listening <00:34:02.722>to <00:34:02.967>me.

559
00:34:03.870 --> 00:34:07.050
<00:34:03.870>If <00:34:03.963>you <00:34:04.056>would <00:34:04.149>like <00:34:04.243>to <00:34:04.336>get <00:34:04.429>in <00:34:04.523>touch <00:34:04.616>at <00:34:04.709>any <00:34:04.803>point, <00:34:04.896>feel <00:34:04.989>free <00:34:05.083>to <00:34:05.176>do <00:34:05.270>so.

560
00:34:08.250 --> 00:34:10.031
<00:34:08.250>I <00:34:08.483>linked <00:34:08.716>a <00:34:08.949>couple <00:34:09.183>of <00:34:09.416>resources <00:34:09.650>here.

561
00:34:10.150 --> 00:34:13.531
<00:34:10.150>I <00:34:10.250>do <00:34:10.350>recommend <00:34:10.450>visiting <00:34:10.550>the <00:34:10.650>official <00:34:10.750>website <00:34:10.850>of <00:34:10.950>RAGAS <00:34:11.050>and <00:34:11.150>have <00:34:11.250>a <00:34:11.350>look <00:34:11.450>at <00:34:11.550>the

562
00:34:13.591 --> 00:34:14.460
documentation.

563
00:34:15.350 --> 00:34:19.130
<00:34:15.350>The <00:34:15.525>original <00:34:15.699>Hugging <00:34:15.875>Face <00:34:16.049>dataset <00:34:16.225>is <00:34:16.400>available <00:34:16.574>for <00:34:16.750>you

564
00:34:19.230 --> 00:34:23.210
<00:34:19.230>to <00:34:19.357>experiment <00:34:19.484>and <00:34:19.611>play <00:34:19.739>with. <00:34:19.866>And <00:34:19.993>if <00:34:20.120>you <00:34:20.248>visit <00:34:20.375>my <00:34:20.502>GitHub <00:34:20.630>profile,

565
00:34:23.410 --> 00:34:27.270
<00:34:23.409>you <00:34:23.585>will <00:34:23.759>see <00:34:23.935>this <00:34:24.109>public <00:34:24.284>repository <00:34:24.460>where <00:34:24.634>I'll <00:34:24.810>store

566
00:34:27.350 --> 00:34:31.210
<00:34:27.350>the <00:34:27.443>slides <00:34:27.536>of <00:34:27.630>today. <00:34:27.723>And <00:34:27.816>of <00:34:27.909>course, <00:34:28.003>you <00:34:28.096>can <00:34:28.190>have <00:34:28.283>a <00:34:28.376>look <00:34:28.469>at <00:34:28.563>the <00:34:28.656>notebook <00:34:28.750>and

567
00:34:31.270 --> 00:34:34.770
<00:34:31.270>run <00:34:31.377>it <00:34:31.485>yourself. <00:34:31.593>Thank <00:34:31.700>you <00:34:31.808>very <00:34:31.916>much, <00:34:32.023>and <00:34:32.131>I <00:34:32.239>wish <00:34:32.346>you <00:34:32.454>a <00:34:32.562>nice <00:34:32.670>day.