WEBVTT

1
00:00:02.545 --> 00:00:03.165
<00:00:02.545>Hi <00:00:03.103>everyone.

2
00:00:03.425 --> 00:00:06.285
<00:00:03.425>So, <00:00:03.599>um, <00:00:03.775>thanks <00:00:03.949>for <00:00:04.125>coming <00:00:04.300>to <00:00:04.475>my, <00:00:04.649>my <00:00:04.824>Talk.

3
00:00:06.285 --> 00:00:08.365
<00:00:06.285>It's <00:00:06.460>actually <00:00:06.635>my <00:00:06.810>first <00:00:06.985>time <00:00:07.160>talking <00:00:07.335>at <00:00:07.510>an <00:00:07.685>international

4
00:00:08.365 --> 00:00:10.245
<00:00:08.365>conference, <00:00:08.598>so <00:00:08.831>thank <00:00:09.065>you <00:00:09.298>for <00:00:09.531>your <00:00:09.765>time.

5
00:00:11.225 --> 00:00:13.405
<00:00:11.225>So <00:00:11.424>today, <00:00:11.625>like <00:00:11.825>the <00:00:12.025>title <00:00:12.225>says, <00:00:12.424>we're <00:00:12.625>going

6
00:00:13.405 --> 00:00:15.085
<00:00:13.405>to <00:00:13.684>be <00:00:13.965>exploring <00:00:14.245>advanced <00:00:14.524>patterns <00:00:14.805>in

7
00:00:15.245 --> 00:00:16.365
<00:00:15.245>retrieval <00:00:15.748>augmented <00:00:16.253>generation.

8
00:00:16.945 --> 00:00:19.165
<00:00:16.945>Uh, <00:00:17.120>and <00:00:17.295>this <00:00:17.470>actually <00:00:17.645>comes <00:00:17.820>from <00:00:17.995>like <00:00:18.170>a <00:00:18.345>year

9
00:00:19.165 --> 00:00:22.205
<00:00:19.165>and <00:00:19.281>a <00:00:19.398>half <00:00:19.515>of <00:00:19.631>work <00:00:19.748>that <00:00:19.865>I <00:00:19.981>did <00:00:20.098>on <00:00:20.215>some <00:00:20.331>of <00:00:20.448>these <00:00:20.564>projects.

10
00:00:22.745 --> 00:00:26.325
<00:00:22.745>Um, <00:00:22.872>and <00:00:22.999>I <00:00:23.126>realized <00:00:23.254>that <00:00:23.381>you <00:00:23.508>always <00:00:23.635>have <00:00:23.763>to <00:00:23.890>go <00:00:24.017>a <00:00:24.145>bit

11
00:00:26.325 --> 00:00:28.445
<00:00:26.325>beyond <00:00:26.605>the <00:00:26.884>basic <00:00:27.165>rag <00:00:27.445>pipeline, <00:00:27.724>um,

12
00:00:28.675 --> 00:00:29.965
<00:00:28.675>that <00:00:28.907>you <00:00:29.139>see <00:00:29.371>on <00:00:29.603>YouTube, <00:00:29.836>right?

13
00:00:29.965 --> 00:00:31.485
<00:00:29.965>The <00:00:30.238>chat <00:00:30.512>with <00:00:30.785>your <00:00:31.059>documents <00:00:31.333>or,

14
00:00:32.105 --> 00:00:33.925
<00:00:32.104>or <00:00:32.455>the, <00:00:32.805>the <00:00:33.154>starter <00:00:33.504>pipeline.

15
00:00:34.705 --> 00:00:37.405
<00:00:34.705>Uh, <00:00:34.812>but <00:00:34.920>before <00:00:35.028>we <00:00:35.135>jump <00:00:35.243>in, <00:00:35.351>uh, <00:00:35.458>who <00:00:35.566>am <00:00:35.674>I? <00:00:35.781>What <00:00:35.889>do <00:00:35.997>I <00:00:36.105>do?

16
00:00:37.835 --> 00:00:41.325
<00:00:37.835>What <00:00:37.975>did <00:00:38.115>I <00:00:38.255>have <00:00:38.395>for <00:00:38.535>breakfast? <00:00:38.675>Uh, <00:00:38.815>my <00:00:38.955>name <00:00:39.095>is <00:00:39.235>caap.

17
00:00:41.385 --> 00:00:43.445
<00:00:41.385>Oh, <00:00:41.525>I <00:00:41.665>had <00:00:41.805>coffee, <00:00:41.945>by <00:00:42.085>the <00:00:42.224>way. <00:00:42.364>My <00:00:42.504>name <00:00:42.644>is <00:00:42.785>caap.

18
00:00:43.585 --> 00:00:45.805
<00:00:43.585>I'm <00:00:43.818>a <00:00:44.051>software <00:00:44.285>developer <00:00:44.518>from <00:00:44.751>Montreal, <00:00:44.985>Quebec,

19
00:00:45.945 --> 00:00:47.085
<00:00:45.945>Canada <00:00:46.971>Earth.

20
00:00:47.505 --> 00:00:49.925
<00:00:47.505>Uh, <00:00:47.645>I've <00:00:47.785>been <00:00:47.925>a <00:00:48.065>software <00:00:48.205>developer <00:00:48.345>for <00:00:48.485>the <00:00:48.625>past <00:00:48.765>10 <00:00:48.905>years,

21
00:00:49.925 --> 00:00:51.445
<00:00:49.925>working <00:00:50.153>on <00:00:50.381>a <00:00:50.609>bunch <00:00:50.836>of <00:00:51.065>different <00:00:51.293>systems.

22
00:00:51.605 --> 00:00:55.085
<00:00:51.605>I <00:00:51.838>worked <00:00:52.071>on <00:00:52.305>content, <00:00:52.538>digitization, <00:00:52.771>financial <00:00:53.004>systems,

23
00:00:55.385 --> 00:00:57.405
<00:00:55.385>search, <00:00:56.085>recommendation, <00:00:56.785>uh,

24
00:00:57.425 --> 00:00:58.805
<00:00:57.425>but <00:00:57.839>today <00:00:58.253>I <00:00:58.667>work

25
00:00:58.825 --> 00:01:02.565
<00:00:58.825>as <00:00:58.965>a <00:00:59.105>tech <00:00:59.245>lead <00:00:59.385>at <00:00:59.525>a <00:00:59.665>worldwide <00:00:59.805>innovation <00:00:59.945>firm <00:01:00.085>called <00:01:00.225>sidia,

26
00:01:02.985 --> 00:01:04.485
<00:01:02.985>uh, <00:01:03.322>based <00:01:03.660>in <00:01:03.997>Montreal, <00:01:04.334>Quebec.

27
00:01:05.385 --> 00:01:08.805
<00:01:05.385>So <00:01:05.525>at <00:01:05.665>Sidia <00:01:05.805>we <00:01:05.945>have <00:01:06.085>a <00:01:06.225>goal <00:01:06.365>of <00:01:06.505>bringing <00:01:06.645>innovative <00:01:06.785>solutions

28
00:01:08.805 --> 00:01:10.405
<00:01:08.805>to <00:01:09.005>life <00:01:09.205>with <00:01:09.405>the <00:01:09.605>help <00:01:09.805>of <00:01:10.005>product <00:01:10.205>design,

29
00:01:11.005 --> 00:01:13.645
<00:01:11.005>software <00:01:11.285>development, <00:01:11.565>artificial <00:01:11.845>intelligence, <00:01:12.125>and <00:01:12.405>robotics.

30
00:01:14.375 --> 00:01:15.845
<00:01:14.375>We're <00:01:14.705>also <00:01:15.036>partners <00:01:15.367>with <00:01:15.698>Mila,

31
00:01:15.845 --> 00:01:17.405
<00:01:15.845>which <00:01:16.125>is <00:01:16.405>the <00:01:16.685>Quebec <00:01:16.965>AI <00:01:17.245>Institute,

32
00:01:18.025 --> 00:01:20.205
<00:01:18.025>and <00:01:18.258>we're <00:01:18.491>solutions <00:01:18.725>partners <00:01:18.958>with <00:01:19.191>Boston <00:01:19.425>Dynamics,

33
00:01:20.205 --> 00:01:21.485
<00:01:20.205>which <00:01:20.369>means <00:01:20.534>that <00:01:20.698>we <00:01:20.863>get <00:01:21.027>to <00:01:21.192>play <00:01:21.357>around

34
00:01:21.485 --> 00:01:23.965
<00:01:21.485>with <00:01:21.660>cool <00:01:21.835>robots <00:01:22.010>like <00:01:22.185>spot <00:01:22.360>that <00:01:22.535>you <00:01:22.710>see <00:01:22.885>here.

35
00:01:24.945 --> 00:01:27.845
<00:01:24.945>Um, <00:01:25.178>this <00:01:25.411>is <00:01:25.645>actually <00:01:25.878>spot <00:01:26.111>from, <00:01:26.345>uh,

36
00:01:27.865 --> 00:01:29.445
<00:01:27.865>two <00:01:28.065>weeks <00:01:28.265>ago <00:01:28.464>we <00:01:28.664>made <00:01:28.865>a <00:01:29.065>draw <00:01:29.265>anything

37
00:01:29.445 --> 00:01:30.885
<00:01:29.445>with <00:01:29.704>diffusion <00:01:29.963>models <00:01:30.222>and <00:01:30.481>G <00:01:30.741>code.

38
00:01:31.265 --> 00:01:32.765
<00:01:31.265>So <00:01:31.457>I <00:01:31.650>always <00:01:31.843>like <00:01:32.036>to <00:01:32.229>plug <00:01:32.422>that <00:01:32.615>in.

39
00:01:33.265 --> 00:01:35.165
<00:01:33.265>Uh, <00:01:33.440>but <00:01:33.615>we <00:01:33.790>don't <00:01:33.965>just <00:01:34.140>play <00:01:34.315>around <00:01:34.490>with <00:01:34.665>robots.

40
00:01:35.385 --> 00:01:36.525
<00:01:35.384>We <00:01:35.641>actually <00:01:35.898>work <00:01:36.154>on <00:01:36.411>tools

41
00:01:36.525 --> 00:01:38.925
<00:01:36.525>that <00:01:36.805>are <00:01:37.085>revolutionizing <00:01:37.365>industries <00:01:37.645>like <00:01:37.925>dentistry,

42
00:01:39.265 --> 00:01:42.485
<00:01:39.265>mining <00:01:39.465>parts, <00:01:39.665>manufacturing, <00:01:39.865>and <00:01:40.065>a <00:01:40.265>list <00:01:40.465>goes <00:01:40.665>on.

43
00:01:42.485 --> 00:01:44.925
<00:01:42.485>So <00:01:42.660>if <00:01:42.835>you're <00:01:43.010>interested, <00:01:43.185>feel <00:01:43.360>free <00:01:43.535>to <00:01:43.710>contact <00:01:43.885>us.

44
00:01:46.355 --> 00:01:50.205
<00:01:46.354>That <00:01:46.510>being <00:01:46.666>said, <00:01:46.821>um, <00:01:46.977>one <00:01:47.132>of <00:01:47.288>the <00:01:47.443>most <00:01:47.599>common <00:01:47.755>problems

45
00:01:50.205 --> 00:01:52.285
<00:01:50.205>that <00:01:50.360>our <00:01:50.516>clients <00:01:50.671>face <00:01:50.827>today <00:01:50.982>has <00:01:51.138>to <00:01:51.293>do <00:01:51.449>with <00:01:51.605>navigating

46
00:01:52.365 --> 00:01:53.845
<00:01:52.365>and <00:01:52.631>researching <00:01:52.897>their <00:01:53.164>own <00:01:53.430>data, <00:01:53.697>right?

47
00:01:53.865 --> 00:01:55.525
<00:01:53.865>And <00:01:54.065>this <00:01:54.265>is <00:01:54.465>where <00:01:54.665>RAG <00:01:54.865>comes <00:01:55.065>into <00:01:55.265>play.

48
00:01:55.945 --> 00:01:58.045
<00:01:55.945>So <00:01:56.100>rag <00:01:56.256>being <00:01:56.411>a <00:01:56.567>way <00:01:56.722>of <00:01:56.878>bringing <00:01:57.033>your <00:01:57.189>own <00:01:57.345>data

49
00:01:58.425 --> 00:02:01.285
<00:01:58.425>to <00:01:58.600>a <00:01:58.774>large <00:01:58.950>language <00:01:59.125>model <00:01:59.300>to <00:01:59.475>extend <00:01:59.649>the <00:01:59.825>context

50
00:02:01.785 --> 00:02:03.285
<00:02:01.785>and <00:02:02.054>give <00:02:02.325>a <00:02:02.595>more <00:02:02.865>grounded <00:02:03.134>response.

51
00:02:04.825 --> 00:02:06.525
<00:02:04.825>So <00:02:05.025>for <00:02:05.225>one <00:02:05.425>last <00:02:05.625>time <00:02:05.825>at <00:02:06.025>this <00:02:06.225>conference,

52
00:02:06.525 --> 00:02:09.205
<00:02:06.525>we're <00:02:06.680>gonna <00:02:06.836>quickly <00:02:06.991>go <00:02:07.147>over <00:02:07.302>the <00:02:07.458>basic <00:02:07.613>pipeline <00:02:07.769>for <00:02:07.925>rag.

53
00:02:09.305 --> 00:02:10.725
<00:02:09.305>I'm <00:02:09.487>sure <00:02:09.670>you've <00:02:09.852>seen <00:02:10.035>a <00:02:10.217>lot <00:02:10.400>of <00:02:10.583>this.

54
00:02:11.265 --> 00:02:12.525
<00:02:11.265>Um, <00:02:11.491>but <00:02:11.718>you <00:02:11.945>have <00:02:12.172>your <00:02:12.399>data

55
00:02:12.525 --> 00:02:14.085
<00:02:12.525>preparation <00:02:12.991>pipeline <00:02:13.458>first, <00:02:13.925>right?

56
00:02:14.445 --> 00:02:16.965
<00:02:14.445>Documents <00:02:14.620>come <00:02:14.795>in, <00:02:14.970>then <00:02:15.144>you <00:02:15.320>split <00:02:15.495>them <00:02:15.670>into <00:02:15.845>chunks,

57
00:02:16.965 --> 00:02:18.965
<00:02:16.965>and <00:02:17.120>then <00:02:17.276>you <00:02:17.431>send <00:02:17.587>those <00:02:17.742>over <00:02:17.898>to <00:02:18.053>an <00:02:18.209>embedding <00:02:18.365>model.

58
00:02:19.545 --> 00:02:22.325
<00:02:19.545>And <00:02:19.700>this <00:02:19.856>embedding <00:02:20.011>model <00:02:20.167>is <00:02:20.322>gonna <00:02:20.478>convert <00:02:20.633>your <00:02:20.789>text <00:02:20.945>into

59
00:02:23.005 --> 00:02:25.245
<00:02:23.005>a <00:02:23.355>semantic <00:02:23.704>representation <00:02:24.055>using <00:02:24.405>vectors.

60
00:02:26.145 --> 00:02:29.165
<00:02:26.145>Uh, <00:02:26.261>so <00:02:26.378>what <00:02:26.495>do <00:02:26.611>we <00:02:26.728>mean <00:02:26.845>by <00:02:26.961>that? <00:02:27.078>So <00:02:27.195>I'll <00:02:27.311>give <00:02:27.428>an <00:02:27.545>example.

61
00:02:29.415 --> 00:02:31.685
<00:02:29.415>Let's <00:02:29.648>say, <00:02:29.881>uh, <00:02:30.114>I <00:02:30.348>have <00:02:30.581>two <00:02:30.815>sentences.

62
00:02:31.745 --> 00:02:34.165
<00:02:31.745>One <00:02:31.861>is <00:02:31.978>I <00:02:32.095>love <00:02:32.211>apples, <00:02:32.328>and <00:02:32.445>the <00:02:32.561>other <00:02:32.678>one <00:02:32.795>is, <00:02:32.911>I <00:02:33.028>love <00:02:33.145>Apple.

63
00:02:34.955 --> 00:02:36.485
<00:02:34.954>Well, <00:02:35.184>they <00:02:35.414>mean <00:02:35.643>two <00:02:35.873>different <00:02:36.102>things, <00:02:36.332>right?

64
00:02:36.545 --> 00:02:37.845
<00:02:36.545>One <00:02:36.712>of <00:02:36.879>'em <00:02:37.046>is <00:02:37.213>talking <00:02:37.380>about <00:02:37.547>a <00:02:37.715>fruit

65
00:02:37.845 --> 00:02:39.325
<00:02:37.845>and <00:02:38.035>the <00:02:38.225>other <00:02:38.415>one's <00:02:38.606>talking <00:02:38.796>about <00:02:38.986>the <00:02:39.177>company.

66
00:02:40.425 --> 00:02:43.565
<00:02:40.425>So <00:02:40.600>these <00:02:40.775>embedding <00:02:40.950>models <00:02:41.125>are, <00:02:41.300>are <00:02:41.475>able <00:02:41.650>to <00:02:41.825>sort

67
00:02:43.565 --> 00:02:44.645
<00:02:43.565>of <00:02:43.889>detect <00:02:44.213>these <00:02:44.537>nuances

68
00:02:44.865 --> 00:02:47.445
<00:02:44.865>and <00:02:45.098>give <00:02:45.331>us <00:02:45.565>semantic <00:02:45.798>representation <00:02:46.031>using <00:02:46.265>vectors.

69
00:02:48.385 --> 00:02:50.605
<00:02:48.385>So <00:02:48.512>once <00:02:48.639>we <00:02:48.766>have <00:02:48.894>our <00:02:49.021>vector, <00:02:49.148>we <00:02:49.275>put <00:02:49.403>in <00:02:49.530>a <00:02:49.657>vector <00:02:49.785>database,

70
00:02:50.745 --> 00:02:52.445
<00:02:50.745>and <00:02:50.945>then <00:02:51.145>we <00:02:51.345>go <00:02:51.545>through <00:02:51.745>the <00:02:51.945>retrieval <00:02:52.145>pipeline.

71
00:02:52.625 --> 00:02:55.805
<00:02:52.625>So <00:02:52.800>ask <00:02:52.975>our <00:02:53.150>question, <00:02:53.325>use <00:02:53.500>the <00:02:53.675>same <00:02:53.850>embedding <00:02:54.025>model

72
00:02:55.865 --> 00:02:57.645
<00:02:55.865>to <00:02:56.215>get <00:02:56.565>a <00:02:56.915>semantic <00:02:57.265>representation,

73
00:02:58.185 --> 00:03:00.325
<00:02:58.185>and <00:02:58.385>then <00:02:58.585>we <00:02:58.785>do <00:02:58.985>approximate <00:02:59.185>nearest <00:02:59.385>neighbor <00:02:59.585>search,

74
00:03:00.325 --> 00:03:02.885
<00:03:00.325>which <00:03:00.500>means <00:03:00.674>that <00:03:00.850>we're <00:03:01.024>gonna <00:03:01.200>find <00:03:01.375>the <00:03:01.549>closest <00:03:01.725>vectors

75
00:03:03.145 --> 00:03:05.405
<00:03:03.145>to <00:03:03.378>our <00:03:03.611>query <00:03:03.845>to <00:03:04.078>get <00:03:04.311>the <00:03:04.545>most

76
00:03:05.605 --> 00:03:07.325
<00:03:05.605>semantically <00:03:06.071>relevant <00:03:06.538>results, <00:03:07.005>right?

77
00:03:07.755 --> 00:03:10.245
<00:03:07.755>Once <00:03:07.882>we <00:03:08.009>have <00:03:08.136>our <00:03:08.264>results, <00:03:08.391>we <00:03:08.518>send <00:03:08.645>it <00:03:08.773>over <00:03:08.900>to <00:03:09.027>an <00:03:09.155>LLM,

78
00:03:10.665 --> 00:03:13.325
<00:03:10.665>and <00:03:10.804>then <00:03:10.945>we <00:03:11.084>can <00:03:11.225>generate <00:03:11.364>a <00:03:11.505>response <00:03:11.644>which <00:03:11.785>will <00:03:11.924>be <00:03:12.065>grounded

79
00:03:13.835 --> 00:03:14.965
<00:03:13.835>with <00:03:14.343>our <00:03:14.852>context.

80
00:03:15.795 --> 00:03:17.165
<00:03:15.795>Okay? <00:03:16.103>So <00:03:16.411>that <00:03:16.719>answers <00:03:17.028>this.

81
00:03:18.585 --> 00:03:20.685
<00:03:18.585>Uh, <00:03:18.785>in <00:03:18.985>the <00:03:19.185>past <00:03:19.385>I <00:03:19.585>had <00:03:19.785>the <00:03:19.985>opportunity

82
00:03:20.685 --> 00:03:22.005
<00:03:20.685>to <00:03:20.854>work <00:03:21.024>on <00:03:21.194>a <00:03:21.363>few <00:03:21.533>of <00:03:21.703>these <00:03:21.873>projects.

83
00:03:22.005 --> 00:03:23.725
<00:03:22.005>So <00:03:22.238>I <00:03:22.471>worked <00:03:22.704>on <00:03:22.938>an <00:03:23.171>RFP <00:03:23.405>assistant.

84
00:03:24.145 --> 00:03:26.485
<00:03:24.145>So <00:03:24.378>we <00:03:24.611>had <00:03:24.845>requests <00:03:25.078>for <00:03:25.311>proposal <00:03:25.545>specialists

85
00:03:26.755 --> 00:03:29.125
<00:03:26.755>that <00:03:26.988>were <00:03:27.221>losing <00:03:27.454>hundreds <00:03:27.688>of <00:03:27.921>hours <00:03:28.155>looking

86
00:03:29.125 --> 00:03:32.085
<00:03:29.125>through <00:03:29.300>the <00:03:29.475>documents, <00:03:29.650>trying <00:03:29.825>to <00:03:30.000>find <00:03:30.175>the <00:03:30.350>right, <00:03:30.525>uh,

87
00:03:32.085 --> 00:03:33.765
<00:03:32.085>information <00:03:32.435>to <00:03:32.785>put <00:03:33.135>in <00:03:33.485>RFPs,

88
00:03:34.505 --> 00:03:36.885
<00:03:34.505>and, <00:03:34.680>um, <00:03:34.855>we <00:03:35.030>just <00:03:35.204>added <00:03:35.380>right <00:03:35.555>on <00:03:35.730>top <00:03:35.905>of

89
00:03:36.885 --> 00:03:37.965
<00:03:36.885>that <00:03:37.128>to <00:03:37.371>solve <00:03:37.614>this <00:03:37.857>problem.

90
00:03:38.505 --> 00:03:40.245
<00:03:38.505>So <00:03:38.738>another <00:03:38.971>one <00:03:39.204>was <00:03:39.438>an <00:03:39.671>engineering <00:03:39.905>assistant.

91
00:03:40.245 --> 00:03:42.525
<00:03:40.245>We <00:03:40.525>had <00:03:40.805>chemical <00:03:41.085>engineers <00:03:41.365>that, <00:03:41.645>again,

92
00:03:42.715 --> 00:03:44.565
<00:03:42.715>they <00:03:42.995>had <00:03:43.275>these <00:03:43.555>thousand <00:03:43.835>page <00:03:44.115>documents,

93
00:03:44.565 --> 00:03:47.005
<00:03:44.565>and <00:03:44.765>they <00:03:44.965>were <00:03:45.165>looking <00:03:45.365>for <00:03:45.565>material <00:03:45.765>specifications, <00:03:45.965>um,

94
00:03:47.185 --> 00:03:50.045
<00:03:47.185>and <00:03:47.385>tolerance <00:03:47.585>levels, <00:03:47.785>and <00:03:47.985>they <00:03:48.185>were <00:03:48.385>losing <00:03:48.585>hundreds

95
00:03:50.045 --> 00:03:52.125
<00:03:50.045>of <00:03:50.278>hours <00:03:50.511>doing <00:03:50.745>this, <00:03:50.978>navigating <00:03:51.211>their <00:03:51.445>documents.

96
00:03:52.625 --> 00:03:53.845
<00:03:52.625>So <00:03:52.899>RAG <00:03:53.174>was <00:03:53.448>a <00:03:53.723>solution.

97
00:03:54.185 --> 00:03:56.605
<00:03:54.185>And <00:03:54.385>then <00:03:54.585>finally <00:03:54.785>we <00:03:54.985>had <00:03:55.185>BI <00:03:55.385>data <00:03:55.585>exploration.

98
00:03:56.625 --> 00:03:58.165
<00:03:56.625>So <00:03:56.971>we <00:03:57.317>added <00:03:57.664>this, <00:03:58.011>uh,

99
00:03:58.195 --> 00:04:01.765
<00:03:58.195>natural <00:03:58.370>language <00:03:58.545>query <00:03:58.720>capabilities <00:03:58.894>on <00:03:59.070>top <00:03:59.245>of <00:03:59.420>BI <00:03:59.595>data.

100
00:04:01.825 --> 00:04:05.325
<00:04:01.825>So <00:04:01.980>we <00:04:02.136>get <00:04:02.291>analyts, <00:04:02.447>um, <00:04:02.602>to <00:04:02.758>explore <00:04:02.913>their <00:04:03.069>own <00:04:03.225>data.

101
00:04:06.625 --> 00:04:10.085
<00:04:06.625>Uh, <00:04:06.765>so <00:04:06.905>while <00:04:07.045>working <00:04:07.185>on <00:04:07.325>three <00:04:07.465>of <00:04:07.605>these <00:04:07.745>projects, <00:04:07.885>I <00:04:08.025>realized

102
00:04:10.085 --> 00:04:13.565
<00:04:10.085>that <00:04:10.225>although <00:04:10.365>they <00:04:10.505>all <00:04:10.645>sort <00:04:10.785>of <00:04:10.925>fit <00:04:11.065>under <00:04:11.205>this <00:04:11.345>umbrella <00:04:11.485>term

103
00:04:13.565 --> 00:04:16.205
<00:04:13.565>of <00:04:13.845>retrieval, <00:04:14.125>augmented <00:04:14.405>generation, <00:04:14.685>the <00:04:14.965>way

104
00:04:16.205 --> 00:04:17.965
<00:04:16.204>that <00:04:16.438>the <00:04:16.671>data <00:04:16.904>was <00:04:17.138>pre-processed, <00:04:17.371>the <00:04:17.604>way

105
00:04:17.965 --> 00:04:20.565
<00:04:17.964>that <00:04:18.120>the <00:04:18.276>data <00:04:18.431>was <00:04:18.587>accessed, <00:04:18.742>it <00:04:18.898>was <00:04:19.053>all <00:04:19.209>completely <00:04:19.364>different.

106
00:04:21.515 --> 00:04:23.535
<00:04:21.515>But <00:04:21.714>they <00:04:21.914>did <00:04:22.115>have <00:04:22.315>some <00:04:22.515>common <00:04:22.715>requirements <00:04:22.914>though,

107
00:04:24.075 --> 00:04:26.535
<00:04:24.075>and <00:04:24.275>the <00:04:24.474>most <00:04:24.675>common <00:04:24.875>ones <00:04:25.075>were <00:04:25.275>one, <00:04:25.474>explainability,

108
00:04:26.955 --> 00:04:29.375
<00:04:26.955>two <00:04:27.305>accuracy, <00:04:27.655>and <00:04:28.005>three <00:04:28.354>complexity.

109
00:04:29.795 --> 00:04:32.215
<00:04:29.795>So <00:04:29.970>they <00:04:30.145>needed <00:04:30.320>to <00:04:30.495>handle <00:04:30.670>these <00:04:30.845>three <00:04:31.020>really <00:04:31.195>well.

110
00:04:32.805 --> 00:04:35.015
<00:04:32.805>When <00:04:33.085>it <00:04:33.365>comes <00:04:33.645>to <00:04:33.925>explainability <00:04:34.205>retrieval,

111
00:04:35.015 --> 00:04:37.735
<00:04:35.015>augmented <00:04:35.294>generation <00:04:35.575>is, <00:04:35.854>uh, <00:04:36.135>it's <00:04:36.414>sort

112
00:04:37.735 --> 00:04:38.935
<00:04:37.735>of <00:04:38.005>an <00:04:38.275>added <00:04:38.545>benefit, <00:04:38.815>right?

113
00:04:39.195 --> 00:04:41.775
<00:04:39.195>You <00:04:39.395>keep <00:04:39.594>track <00:04:39.795>of <00:04:39.995>the, <00:04:40.195>the, <00:04:40.395>the <00:04:40.594>documents

114
00:04:41.775 --> 00:04:43.335
<00:04:41.775>that <00:04:41.974>you <00:04:42.174>use <00:04:42.375>when <00:04:42.575>you're <00:04:42.775>answering <00:04:42.974>a <00:04:43.174>query.

115
00:04:43.515 --> 00:04:45.775
<00:04:43.515>So <00:04:43.748>this <00:04:43.981>one <00:04:44.215>handles <00:04:44.448>itself <00:04:44.681>pretty <00:04:44.914>well

116
00:04:46.005 --> 00:04:47.255
<00:04:46.005>when <00:04:46.286>it <00:04:46.567>comes <00:04:46.848>to <00:04:47.130>accuracy.

117
00:04:47.915 --> 00:04:51.895
<00:04:47.915>You <00:04:48.070>can <00:04:48.226>ask <00:04:48.381>hundreds <00:04:48.537>of <00:04:48.692>different <00:04:48.848>questions <00:04:49.003>to <00:04:49.159>improve, <00:04:49.315>uh,

118
00:04:51.895 --> 00:04:52.975
<00:04:51.895>your <00:04:52.219>search <00:04:52.543>results, <00:04:52.867>right?

119
00:04:53.395 --> 00:04:56.295
<00:04:53.395>And <00:04:53.534>we're <00:04:53.674>gonna <00:04:53.815>look <00:04:53.955>into <00:04:54.094>those <00:04:54.234>in <00:04:54.375>just <00:04:54.515>a <00:04:54.655>few <00:04:54.794>seconds.

120
00:04:56.395 --> 00:04:58.975
<00:04:56.395>But <00:04:56.534>you <00:04:56.674>can <00:04:56.815>ask <00:04:56.955>like, <00:04:57.094>which <00:04:57.234>embedding <00:04:57.375>model <00:04:57.515>do <00:04:57.655>I <00:04:57.794>use?

121
00:04:58.985 --> 00:05:00.935
<00:04:58.985>Which, <00:04:59.185>uh, <00:04:59.385>how <00:04:59.585>do <00:04:59.785>I <00:04:59.985>chunk <00:05:00.185>my <00:05:00.385>text?

122
00:05:01.275 --> 00:05:03.135
<00:05:01.275>Should <00:05:01.508>I <00:05:01.741>quit <00:05:01.974>and <00:05:02.208>become <00:05:02.441>a <00:05:02.674>monk?

123
00:05:03.395 --> 00:05:05.495
<00:05:03.395>Or, <00:05:03.570>you <00:05:03.745>know, <00:05:03.919>hundreds <00:05:04.094>of <00:05:04.270>questions <00:05:04.445>you <00:05:04.620>can <00:05:04.794>ask.

124
00:05:06.645 --> 00:05:08.375
<00:05:06.645>Next <00:05:07.111>up, <00:05:07.578>uh, <00:05:08.044>complexity,

125
00:05:08.375 --> 00:05:09.895
<00:05:08.375>and <00:05:08.570>this <00:05:08.765>is <00:05:08.961>where <00:05:09.156>things <00:05:09.352>get <00:05:09.547>really <00:05:09.743>fun.

126
00:05:10.165 --> 00:05:12.695
<00:05:10.165>Okay? <00:05:10.320>So <00:05:10.476>this <00:05:10.631>is <00:05:10.787>where <00:05:10.942>we <00:05:11.098>start <00:05:11.253>getting <00:05:11.409>into <00:05:11.565>reasoning

127
00:05:13.075 --> 00:05:14.895
<00:05:13.075>and <00:05:13.354>building <00:05:13.635>agentic <00:05:13.914>rag, <00:05:14.195>which <00:05:14.474>is

128
00:05:14.895 --> 00:05:16.095
<00:05:14.895>what <00:05:15.111>we're <00:05:15.327>gonna <00:05:15.543>touch <00:05:15.759>on <00:05:15.975>today.

129
00:05:18.275 --> 00:05:21.015
<00:05:18.275>So <00:05:18.474>we're <00:05:18.674>just <00:05:18.875>gonna <00:05:19.075>go <00:05:19.275>over <00:05:19.474>some, <00:05:19.674>uh,

130
00:05:21.535 --> 00:05:22.775
<00:05:21.535>accuracy <00:05:21.758>improvements <00:05:21.981>that <00:05:22.204>you <00:05:22.427>can <00:05:22.650>do.

131
00:05:22.875 --> 00:05:24.295
<00:05:22.875>So <00:05:23.088>one <00:05:23.301>of <00:05:23.514>'em <00:05:23.727>is <00:05:23.940>search <00:05:24.153>improvements.

132
00:05:24.795 --> 00:05:26.495
<00:05:24.795>Uh, <00:05:25.075>and <00:05:25.355>the <00:05:25.635>most <00:05:25.915>basic <00:05:26.195>example

133
00:05:26.515 --> 00:05:28.015
<00:05:26.515>of <00:05:26.784>this <00:05:27.055>is <00:05:27.325>through <00:05:27.594>hybrid <00:05:27.865>search.

134
00:05:28.475 --> 00:05:31.735
<00:05:28.475>So <00:05:28.650>you <00:05:28.825>have <00:05:29.000>your <00:05:29.175>exact <00:05:29.350>same, <00:05:29.525>uh, <00:05:29.700>query <00:05:29.875>pipeline,

135
00:05:32.075 --> 00:05:33.895
<00:05:32.075>but <00:05:32.275>this <00:05:32.474>time <00:05:32.675>we're <00:05:32.875>doing <00:05:33.075>a <00:05:33.275>secondary <00:05:33.474>search,

136
00:05:33.895 --> 00:05:35.255
<00:05:33.895>which <00:05:34.139>is <00:05:34.384>a, <00:05:34.629>uh, <00:05:34.874>keyword <00:05:35.119>search.

137
00:05:35.275 --> 00:05:37.015
<00:05:35.275>So <00:05:35.625>think <00:05:35.974>bn <00:05:36.325>BN <00:05:36.674>25.

138
00:05:37.635 --> 00:05:39.415
<00:05:37.635>Um, <00:05:37.914>and <00:05:38.195>then <00:05:38.474>with <00:05:38.755>those <00:05:39.034>results,

139
00:05:39.415 --> 00:05:41.855
<00:05:39.415>we're <00:05:39.590>gonna <00:05:39.765>mix <00:05:39.940>them <00:05:40.115>together <00:05:40.290>with <00:05:40.465>a <00:05:40.640>reran, <00:05:40.815>okay?

140
00:05:43.115 --> 00:05:45.695
<00:05:43.115>And <00:05:43.270>by <00:05:43.426>doing <00:05:43.581>that, <00:05:43.737>we're <00:05:43.892>getting <00:05:44.048>the <00:05:44.203>most <00:05:44.359>relevant <00:05:44.515>results

141
00:05:45.895 --> 00:05:49.175
<00:05:45.895>from <00:05:46.034>both <00:05:46.174>searches <00:05:46.315>mixed <00:05:46.455>together <00:05:46.594>so <00:05:46.734>that <00:05:46.875>we <00:05:47.015>can <00:05:47.155>send <00:05:47.294>it

142
00:05:49.175 --> 00:05:50.695
<00:05:49.175>to <00:05:49.403>an <00:05:49.631>LLM <00:05:49.859>to <00:05:50.087>generate <00:05:50.315>a <00:05:50.543>response.

143
00:05:51.275 --> 00:05:53.415
<00:05:51.275>So <00:05:51.474>I <00:05:51.674>pulled <00:05:51.875>an <00:05:52.075>example <00:05:52.275>from <00:05:52.474>the, <00:05:52.674>uh,

144
00:05:53.475 --> 00:05:55.975
<00:05:53.475>the <00:05:53.650>engineering <00:05:53.825>rag <00:05:54.000>pipeline <00:05:54.175>that <00:05:54.350>I, <00:05:54.525>I <00:05:54.700>talked <00:05:54.875>about.

145
00:05:56.395 --> 00:05:58.085
<00:05:56.395>So <00:05:56.674>what <00:05:56.955>are <00:05:57.234>the <00:05:57.515>material <00:05:57.794>specifications

146
00:05:58.105 --> 00:06:00.325
<00:05:58.105>for <00:05:58.280>H <00:05:58.455>3 <00:05:58.630>3, <00:05:58.805>5, <00:05:58.980>6, <00:05:59.155>0 <00:05:59.330>0 <00:05:59.505>7?

147
00:06:00.795 --> 00:06:03.525
<00:06:00.795>Well, <00:06:00.970>that <00:06:01.145>last <00:06:01.320>part <00:06:01.495>has <00:06:01.670>no <00:06:01.845>semantic <00:06:02.020>meaning, <00:06:02.195>right?

148
00:06:04.105 --> 00:06:05.525
<00:06:04.105>But <00:06:04.318>if <00:06:04.531>you're <00:06:04.744>doing <00:06:04.957>a <00:06:05.169>keyword <00:06:05.383>search,

149
00:06:05.525 --> 00:06:06.725
<00:06:05.525>you're <00:06:05.741>actually <00:06:05.957>gotta <00:06:06.173>pick <00:06:06.389>it <00:06:06.605>up.

150
00:06:07.705 --> 00:06:09.845
<00:06:07.705>So <00:06:07.860>this <00:06:08.016>is <00:06:08.171>just <00:06:08.327>one <00:06:08.482>way <00:06:08.638>of <00:06:08.793>improving <00:06:08.949>your <00:06:09.104>accuracy.

151
00:06:11.275 --> 00:06:14.245
<00:06:11.275>Another <00:06:11.430>thing <00:06:11.586>we <00:06:11.741>can <00:06:11.897>do, <00:06:12.052>uh, <00:06:12.208>is <00:06:12.363>post <00:06:12.519>retrieval <00:06:12.674>improvements.

152
00:06:14.425 --> 00:06:17.205
<00:06:14.425>So <00:06:14.625>we <00:06:14.825>just <00:06:15.025>talked <00:06:15.225>about <00:06:15.425>reciprocal <00:06:15.625>rank <00:06:15.825>fusion

153
00:06:17.265 --> 00:06:18.445
<00:06:17.265>and <00:06:17.619>using <00:06:17.973>re <00:06:18.327>rankers.

154
00:06:18.825 --> 00:06:20.325
<00:06:18.825>Uh, <00:06:19.017>but <00:06:19.210>another <00:06:19.403>thing <00:06:19.596>that <00:06:19.789>you <00:06:19.982>could <00:06:20.175>do

155
00:06:20.325 --> 00:06:21.445
<00:06:20.325>is <00:06:20.577>going <00:06:20.829>small <00:06:21.081>to <00:06:21.332>big.

156
00:06:22.025 --> 00:06:25.205
<00:06:22.025>So <00:06:22.180>in <00:06:22.336>this <00:06:22.491>scenario, <00:06:22.647>we're <00:06:22.802>getting <00:06:22.958>our, <00:06:23.113>our <00:06:23.269>documents, <00:06:23.424>uh,

157
00:06:25.265 --> 00:06:27.885
<00:06:25.265>but <00:06:25.420>we're, <00:06:25.576>we're <00:06:25.731>also <00:06:25.887>going <00:06:26.042>to <00:06:26.198>do <00:06:26.353>a <00:06:26.509>second <00:06:26.664>step

158
00:06:28.145 --> 00:06:30.805
<00:06:28.145>of <00:06:28.424>retrieving, <00:06:28.705>retrieving <00:06:28.984>the <00:06:29.265>sibling <00:06:29.544>nodes.

159
00:06:30.955 --> 00:06:34.125
<00:06:30.955>Okay? <00:06:31.094>So <00:06:31.234>we're <00:06:31.375>gonna <00:06:31.515>expand <00:06:31.655>on <00:06:31.794>the <00:06:31.935>context <00:06:32.075>of <00:06:32.215>a <00:06:32.354>chunk

160
00:06:34.125 --> 00:06:36.005
<00:06:34.125>that <00:06:34.358>we <00:06:34.591>found <00:06:34.825>with <00:06:35.058>our <00:06:35.291>vector <00:06:35.525>search.

161
00:06:37.195 --> 00:06:39.255
<00:06:37.195>If <00:06:37.474>you <00:06:37.755>have, <00:06:38.034>uh, <00:06:38.315>parent <00:06:38.594>summarizations,

162
00:06:39.315 --> 00:06:40.335
<00:06:39.315>you <00:06:39.498>can <00:06:39.682>do <00:06:39.865>the <00:06:40.049>same <00:06:40.233>thing.

163
00:06:40.595 --> 00:06:42.215
<00:06:40.595>Uh, <00:06:40.875>instead <00:06:41.155>of <00:06:41.435>getting <00:06:41.715>sibling <00:06:41.995>nodes,

164
00:06:42.215 --> 00:06:43.655
<00:06:42.215>you <00:06:42.539>can <00:06:42.863>get <00:06:43.187>parent <00:06:43.510>nodes.

165
00:06:44.515 --> 00:06:47.215
<00:06:44.515>So <00:06:44.670>this <00:06:44.826>is <00:06:44.981>just <00:06:45.137>increasing <00:06:45.292>the <00:06:45.448>context <00:06:45.603>for <00:06:45.759>our <00:06:45.914>LLM,

166
00:06:47.215 --> 00:06:49.375
<00:06:47.215>so <00:06:47.448>you <00:06:47.681>could <00:06:47.915>answer <00:06:48.148>the <00:06:48.381>question <00:06:48.615>better.

167
00:06:51.365 --> 00:06:53.695
<00:06:51.365>Another <00:06:51.540>thing <00:06:51.715>we <00:06:51.890>can <00:06:52.065>do <00:06:52.240>is <00:06:52.415>data <00:06:52.590>preparation <00:06:52.765>improvements.

168
00:06:53.695 --> 00:06:55.095
<00:06:53.695>And <00:06:54.010>data <00:06:54.325>preparation <00:06:54.640>is <00:06:54.955>huge.

169
00:06:55.515 --> 00:06:57.855
<00:06:55.515>Um, <00:06:55.690>so <00:06:55.865>in <00:06:56.039>this <00:06:56.215>case, <00:06:56.390>what <00:06:56.565>we're <00:06:56.740>doing <00:06:56.914>is

170
00:06:58.415 --> 00:07:01.735
<00:06:58.415>recursive <00:06:58.695>abstractive <00:06:58.975>processing <00:06:59.255>free <00:06:59.535>organized <00:06:59.815>retrieval,

171
00:07:02.065 --> 00:07:03.775
<00:07:02.065>which <00:07:02.240>you <00:07:02.415>only <00:07:02.590>wanna <00:07:02.765>say <00:07:02.940>once <00:07:03.115>in <00:07:03.290>your <00:07:03.465>life,

172
00:07:03.855 --> 00:07:05.535
<00:07:03.855>I <00:07:04.030>think <00:07:04.205>they <00:07:04.380>wanted <00:07:04.555>the <00:07:04.730>cool <00:07:04.905>name <00:07:05.080>of <00:07:05.255>Raptor.

173
00:07:05.995 --> 00:07:08.975
<00:07:05.995>Uh, <00:07:06.135>but <00:07:06.275>what <00:07:06.415>you're <00:07:06.555>doing <00:07:06.695>here <00:07:06.835>is <00:07:06.975>you're <00:07:07.115>sort <00:07:07.255>of <00:07:07.395>recursively

174
00:07:09.855 --> 00:07:11.815
<00:07:09.855>summarizing <00:07:10.135>your <00:07:10.415>documents <00:07:10.695>until <00:07:10.975>you <00:07:11.255>get

175
00:07:11.815 --> 00:07:13.855
<00:07:11.815>to <00:07:12.048>one <00:07:12.281>parent <00:07:12.515>document <00:07:12.748>that's <00:07:12.981>summarizing <00:07:13.215>everything,

176
00:07:14.595 --> 00:07:16.375
<00:07:14.595>and <00:07:14.875>you're <00:07:15.155>effectively <00:07:15.435>augmenting <00:07:15.715>your <00:07:15.995>data.

177
00:07:16.395 --> 00:07:19.655
<00:07:16.395>You're <00:07:16.550>creating <00:07:16.706>new <00:07:16.861>data <00:07:17.017>around <00:07:17.172>the <00:07:17.328>chunks <00:07:17.483>that <00:07:17.639>you <00:07:17.794>have

178
00:07:20.715 --> 00:07:24.415
<00:07:20.715>to <00:07:20.890>increase <00:07:21.065>your <00:07:21.239>chance <00:07:21.414>of <00:07:21.590>getting <00:07:21.765>semantic, <00:07:21.940>uh, <00:07:22.114>similar,

179
00:07:24.635 --> 00:07:26.175
<00:07:24.635>uh, <00:07:24.981>documents <00:07:25.328>or <00:07:25.674>good <00:07:26.021>results.

180
00:07:26.995 --> 00:07:29.055
<00:07:26.995>So <00:07:27.195>we <00:07:27.395>can <00:07:27.595>go <00:07:27.795>on <00:07:27.995>for <00:07:28.195>a <00:07:28.395>while.

181
00:07:29.055 --> 00:07:31.255
<00:07:29.055>There's <00:07:29.210>a <00:07:29.366>bunch <00:07:29.521>of, <00:07:29.677>uh, <00:07:29.832>different <00:07:29.988>improvements <00:07:30.143>you <00:07:30.299>can <00:07:30.455>do,

182
00:07:31.255 --> 00:07:34.295
<00:07:31.255>like <00:07:31.410>these, <00:07:31.566>you <00:07:31.721>can <00:07:31.877>even <00:07:32.032>replace <00:07:32.188>the <00:07:32.343>whole <00:07:32.499>thing <00:07:32.655>with,

183
00:07:34.315 --> 00:07:35.455
<00:07:34.315>uh, <00:07:34.657>a <00:07:34.998>graph, <00:07:35.341>right?

184
00:07:35.515 --> 00:07:38.575
<00:07:35.515>We <00:07:35.631>saw <00:07:35.748>NEO <00:07:35.865>four <00:07:35.981>J <00:07:36.098>do <00:07:36.215>a, <00:07:36.331>a <00:07:36.448>nice <00:07:36.565>presentation <00:07:36.681>on <00:07:36.798>graph <00:07:36.914>rag.

185
00:07:38.875 --> 00:07:40.015
<00:07:38.875>Uh, <00:07:39.046>you <00:07:39.217>can <00:07:39.388>do <00:07:39.558>that <00:07:39.730>as <00:07:39.901>well,

186
00:07:40.275 --> 00:07:41.815
<00:07:40.275>but <00:07:41.661>ultimately,

187
00:07:42.655 --> 00:07:44.855
<00:07:42.655>a <00:07:42.830>simple <00:07:43.005>rack <00:07:43.179>pipeline <00:07:43.354>is <00:07:43.530>never <00:07:43.705>gonna <00:07:43.880>be <00:07:44.054>enough.

188
00:07:45.755 --> 00:07:50.135
<00:07:45.755>So <00:07:45.910>in <00:07:46.066>some <00:07:46.221>cases, <00:07:46.377>you <00:07:46.532>need <00:07:46.688>to <00:07:46.843>add <00:07:46.999>some <00:07:47.155>reasoning.

189
00:07:50.195 --> 00:07:52.055
<00:07:50.195>You <00:07:50.395>need <00:07:50.594>to <00:07:50.795>work <00:07:50.995>towards <00:07:51.195>the <00:07:51.395>goal, <00:07:51.594>right?

190
00:07:52.715 --> 00:07:53.775
<00:07:52.715>So <00:07:53.033>sometimes <00:07:53.351>you <00:07:53.669>need

191
00:07:53.775 --> 00:07:55.855
<00:07:53.775>to <00:07:53.974>break <00:07:54.174>down <00:07:54.375>a <00:07:54.575>question <00:07:54.775>into <00:07:54.974>multiple <00:07:55.174>questions

192
00:07:56.155 --> 00:07:57.695
<00:07:56.155>and <00:07:56.432>look <00:07:56.709>into <00:07:56.986>different <00:07:57.263>data <00:07:57.541>sources.

193
00:07:58.675 --> 00:08:01.855
<00:07:58.675>And <00:07:58.815>this <00:07:58.955>is <00:07:59.095>where <00:07:59.235>Ag <00:07:59.375>Agentic <00:07:59.515>rag <00:07:59.655>comes <00:07:59.795>into <00:07:59.935>play, <00:08:00.075>okay?

194
00:08:02.915 --> 00:08:07.165
<00:08:02.915>So <00:08:03.055>you <00:08:03.195>probably, <00:08:03.335>um, <00:08:03.475>you <00:08:03.615>probably <00:08:03.755>already <00:08:03.895>heard <00:08:04.035>of <00:08:04.175>auto <00:08:04.315>GPT

195
00:08:07.165 --> 00:08:09.085
<00:08:07.165>before <00:08:07.398>it <00:08:07.631>came <00:08:07.865>out <00:08:08.098>like <00:08:08.331>a <00:08:08.565>year

196
00:08:09.085 --> 00:08:12.405
<00:08:09.085>and <00:08:09.212>a <00:08:09.339>half <00:08:09.466>ago, <00:08:09.594>um, <00:08:09.721>roughly <00:08:09.848>at <00:08:09.975>the <00:08:10.103>same <00:08:10.230>time <00:08:10.357>as <00:08:10.484>GPT-4.

197
00:08:14.305 --> 00:08:17.445
<00:08:14.305>And <00:08:14.480>if <00:08:14.655>you <00:08:14.830>use <00:08:15.005>it, <00:08:15.180>you <00:08:15.355>probably <00:08:15.530>realize <00:08:15.705>that,

198
00:08:17.665 --> 00:08:18.965
<00:08:17.665>uh, <00:08:17.957>it <00:08:18.250>didn't <00:08:18.542>do <00:08:18.835>much.

199
00:08:19.345 --> 00:08:22.245
<00:08:19.345>So <00:08:19.500>it <00:08:19.656>was <00:08:19.811>the <00:08:19.967>first <00:08:20.122>real <00:08:20.278>agent <00:08:20.433>that <00:08:20.589>came <00:08:20.745>out,

200
00:08:22.385 --> 00:08:25.245
<00:08:22.385>and <00:08:22.560>it <00:08:22.735>had <00:08:22.909>a <00:08:23.085>really <00:08:23.260>interesting <00:08:23.435>approach <00:08:23.610>to <00:08:23.784>reasoning,

201
00:08:25.665 --> 00:08:28.085
<00:08:25.665>but <00:08:25.820>it <00:08:25.976>sort <00:08:26.131>of <00:08:26.287>got <00:08:26.442>stuck <00:08:26.598>in <00:08:26.753>a <00:08:26.909>loop, <00:08:27.065>uh,

202
00:08:28.085 --> 00:08:29.085
<00:08:28.085>when <00:08:28.265>you <00:08:28.445>asked <00:08:28.625>it <00:08:28.805>a <00:08:28.984>question

203
00:08:29.225 --> 00:08:30.805
<00:08:29.225>or <00:08:29.575>it <00:08:29.925>never <00:08:30.275>really <00:08:30.625>got

204
00:08:30.805 --> 00:08:32.045
<00:08:30.805>to <00:08:30.964>the <00:08:31.123>goal <00:08:31.283>that <00:08:31.442>you <00:08:31.602>were <00:08:31.761>looking <00:08:31.920>for.

205
00:08:32.905 --> 00:08:34.925
<00:08:32.905>But <00:08:33.060>you <00:08:33.216>fast <00:08:33.371>forward <00:08:33.527>a <00:08:33.682>year <00:08:33.838>and <00:08:33.993>a <00:08:34.149>half, <00:08:34.304>uh,

206
00:08:34.945 --> 00:08:36.685
<00:08:34.945>and <00:08:35.225>LLMs <00:08:35.505>are <00:08:35.785>getting <00:08:36.065>better, <00:08:36.345>uh,

207
00:08:36.685 --> 00:08:38.765
<00:08:36.684>they're <00:08:36.964>getting <00:08:37.244>cheaper, <00:08:37.525>they're <00:08:37.804>getting <00:08:38.084>faster.

208
00:08:39.465 --> 00:08:42.205
<00:08:39.465>Um, <00:08:39.815>and <00:08:40.165>generally <00:08:40.515>we're <00:08:40.865>learning

209
00:08:42.205 --> 00:08:43.485
<00:08:42.205>to <00:08:42.435>work <00:08:42.665>with <00:08:42.896>them <00:08:43.126>more <00:08:43.357>efficiently.

210
00:08:43.825 --> 00:08:46.405
<00:08:43.825>So <00:08:43.980>this <00:08:44.136>is <00:08:44.291>why <00:08:44.447>most <00:08:44.602>industry <00:08:44.758>leaders <00:08:44.913>are <00:08:45.069>actually <00:08:45.225>adopting

211
00:08:46.795 --> 00:08:47.885
<00:08:46.795>agentic <00:08:47.776>workflows

212
00:08:48.025 --> 00:08:49.885
<00:08:48.025>and <00:08:48.258>they're <00:08:48.491>building <00:08:48.725>their <00:08:48.958>own <00:08:49.191>agentic <00:08:49.425>frameworks.

213
00:08:51.095 --> 00:08:53.075
<00:08:51.095>So <00:08:51.270>what <00:08:51.445>do <00:08:51.620>I <00:08:51.795>mean <00:08:51.970>by <00:08:52.145>agentic <00:08:52.320>rag <00:08:52.495>systems?

214
00:08:53.855 --> 00:08:56.405
<00:08:53.855>So <00:08:54.055>these <00:08:54.255>are, <00:08:54.455>uh, <00:08:54.655>systems <00:08:54.855>that <00:08:55.055>make <00:08:55.255>use

215
00:08:56.405 --> 00:08:57.725
<00:08:56.405>of <00:08:56.702>a <00:08:56.999>few <00:08:57.296>key <00:08:57.593>components.

216
00:08:58.185 --> 00:09:00.605
<00:08:58.185>So <00:08:58.324>one <00:08:58.464>of <00:08:58.604>them <00:08:58.744>is, <00:08:58.885>uh, <00:08:59.025>they <00:08:59.165>make <00:08:59.305>use <00:08:59.444>of <00:08:59.584>tools,

217
00:09:00.745 --> 00:09:02.485
<00:09:00.745>so <00:09:01.211>function <00:09:01.678>calling, <00:09:02.145>uh,

218
00:09:02.545 --> 00:09:04.605
<00:09:02.545>but <00:09:02.719>they'll <00:09:02.895>also <00:09:03.069>be <00:09:03.245>able <00:09:03.420>to <00:09:03.594>recall <00:09:03.770>with <00:09:03.944>memory,

219
00:09:04.915 --> 00:09:06.565
<00:09:04.915>they'll <00:09:05.194>be <00:09:05.474>able <00:09:05.755>to <00:09:06.035>plan <00:09:06.314>effectively,

220
00:09:06.665 --> 00:09:08.845
<00:09:06.665>and <00:09:06.792>they'll <00:09:06.919>be <00:09:07.046>able <00:09:07.174>to <00:09:07.301>reflect <00:09:07.428>on <00:09:07.555>what <00:09:07.683>they're <00:09:07.810>about <00:09:07.937>to <00:09:08.064>say.

221
00:09:09.945 --> 00:09:11.405
<00:09:09.945>And <00:09:10.164>at <00:09:10.383>the <00:09:10.602>very <00:09:10.821>core <00:09:11.040>of <00:09:11.259>everything,

222
00:09:11.895 --> 00:09:13.645
<00:09:11.895>we're <00:09:12.175>using <00:09:12.454>a <00:09:12.735>large <00:09:13.015>language <00:09:13.295>model

223
00:09:13.705 --> 00:09:14.925
<00:09:13.705>to <00:09:13.979>orchestrate <00:09:14.254>the <00:09:14.528>whole <00:09:14.803>thing.

224
00:09:15.515 --> 00:09:20.275
<00:09:15.515>Okay? <00:09:15.689>So <00:09:15.865>one <00:09:16.040>of <00:09:16.215>my <00:09:16.390>favorite <00:09:16.564>papers <00:09:16.740>out <00:09:16.915>there

225
00:09:20.415 --> 00:09:23.715
<00:09:20.415>for, <00:09:20.589>uh, <00:09:20.765>an <00:09:20.939>ENT <00:09:21.115>system <00:09:21.290>is <00:09:21.464>the <00:09:21.640>Eureka <00:09:21.814>paper.

226
00:09:24.295 --> 00:09:27.555
<00:09:24.295>So <00:09:24.435>what <00:09:24.574>you're <00:09:24.714>looking <00:09:24.854>at <00:09:24.995>here <00:09:25.135>is <00:09:25.275>not <00:09:25.415>a <00:09:25.555>robot <00:09:25.694>powered

227
00:09:27.555 --> 00:09:30.075
<00:09:27.555>by <00:09:27.694>an <00:09:27.834>LLM, <00:09:27.974>but <00:09:28.114>it's <00:09:28.255>actually <00:09:28.395>a <00:09:28.535>robot <00:09:28.675>that's <00:09:28.814>been <00:09:28.954>trained

228
00:09:30.735 --> 00:09:33.555
<00:09:30.735>to <00:09:30.910>balance <00:09:31.085>on <00:09:31.260>a <00:09:31.435>yoga <00:09:31.610>ball <00:09:31.785>by <00:09:31.960>an <00:09:32.135>LLM.

229
00:09:33.775 --> 00:09:35.835
<00:09:33.775>Uh, <00:09:33.975>and <00:09:34.175>they <00:09:34.375>did <00:09:34.574>that <00:09:34.775>through <00:09:34.975>reinforcement <00:09:35.175>learning.

230
00:09:36.695 --> 00:09:38.995
<00:09:36.695>So <00:09:36.895>when <00:09:37.095>you're <00:09:37.295>doing <00:09:37.495>reinforcement <00:09:37.695>learning, <00:09:37.895>you <00:09:38.095>have

231
00:09:38.995 --> 00:09:40.115
<00:09:38.995>to <00:09:39.247>shape <00:09:39.499>a <00:09:39.751>reward <00:09:40.003>function.

232
00:09:40.655 --> 00:09:42.355
<00:09:40.655>Uh, <00:09:40.888>and <00:09:41.121>a <00:09:41.355>reward <00:09:41.588>function <00:09:41.821>will <00:09:42.055>tell

233
00:09:42.495 --> 00:09:44.675
<00:09:42.495>how <00:09:42.670>your <00:09:42.845>subject <00:09:43.020>does <00:09:43.195>at <00:09:43.370>a <00:09:43.545>specific <00:09:43.720>task, <00:09:43.895>right?

234
00:09:44.695 --> 00:09:45.955
<00:09:44.695>It, <00:09:44.921>it'll <00:09:45.148>give <00:09:45.375>it <00:09:45.602>a <00:09:45.829>score.

235
00:09:46.215 --> 00:09:48.395
<00:09:46.215>Um, <00:09:46.415>and <00:09:46.615>the <00:09:46.815>better <00:09:47.015>the <00:09:47.215>score, <00:09:47.415>you <00:09:47.615>know,

236
00:09:48.415 --> 00:09:51.115
<00:09:48.415>the <00:09:48.570>be <00:09:48.726>the <00:09:48.881>better <00:09:49.037>the <00:09:49.192>candidate <00:09:49.348>for <00:09:49.503>the <00:09:49.659>next <00:09:49.814>phase.

237
00:09:52.935 --> 00:09:56.755
<00:09:52.935>So <00:09:53.074>what <00:09:53.214>the <00:09:53.354>LLM <00:09:53.494>did <00:09:53.635>here <00:09:53.775>is <00:09:53.915>it <00:09:54.055>randomized <00:09:54.194>some <00:09:54.334>parameters,

238
00:09:57.135 --> 00:09:58.755
<00:09:57.135>uh, <00:09:57.485>created <00:09:57.835>that <00:09:58.185>reward <00:09:58.535>function,

239
00:09:59.055 --> 00:10:01.835
<00:09:59.055>and <00:09:59.210>it <00:09:59.366>worked <00:09:59.521>hundreds <00:09:59.677>of <00:09:59.832>different <00:09:59.988>times <00:10:00.143>in <00:10:00.299>parallel <00:10:00.454>just

240
00:10:01.835 --> 00:10:03.195
<00:10:01.835>to <00:10:02.039>hone <00:10:02.243>in <00:10:02.447>on <00:10:02.651>the <00:10:02.855>best <00:10:03.059>solution

241
00:10:04.125 --> 00:10:06.385
<00:10:04.125>to <00:10:04.300>get <00:10:04.475>a <00:10:04.650>robot <00:10:04.825>balancing <00:10:05.000>on <00:10:05.175>a <00:10:05.350>yoga <00:10:05.525>ball.

242
00:10:06.525 --> 00:10:08.585
<00:10:06.525>It <00:10:06.680>actually <00:10:06.836>did <00:10:06.991>better <00:10:07.147>than <00:10:07.302>a <00:10:07.458>human <00:10:07.613>in <00:10:07.769>this, <00:10:07.925>uh,

243
00:10:08.585 --> 00:10:12.145
<00:10:08.585>in <00:10:08.760>this <00:10:08.935>scenario, <00:10:09.110>just <00:10:09.285>because <00:10:09.460>LLMs <00:10:09.635>are <00:10:09.810>infinitely <00:10:09.985>patient.

244
00:10:13.365 --> 00:10:18.265
<00:10:13.365>So, <00:10:13.645>really <00:10:13.925>cool <00:10:14.205>stuff. <00:10:14.485>Let's <00:10:14.765>jump

245
00:10:18.295 --> 00:10:20.345
<00:10:18.295>into <00:10:18.645>agentic <00:10:18.995>rag <00:10:19.344>patterns, <00:10:19.694>right?

246
00:10:20.925 --> 00:10:23.865
<00:10:20.925>So <00:10:21.125>first <00:10:21.324>one <00:10:21.525>is, <00:10:21.724>uh, <00:10:21.925>self-critique <00:10:22.125>and <00:10:22.324>reflection.

247
00:10:24.845 --> 00:10:26.305
<00:10:24.845>We <00:10:25.032>all <00:10:25.220>do <00:10:25.408>it <00:10:25.595>as <00:10:25.783>human <00:10:25.971>beings, <00:10:26.159>right?

248
00:10:26.305 --> 00:10:30.625
<00:10:26.305>When <00:10:26.432>we, <00:10:26.559>uh, <00:10:26.686>we, <00:10:26.814>we <00:10:26.941>think <00:10:27.068>we <00:10:27.195>don't <00:10:27.323>just <00:10:27.450>blurt <00:10:27.577>out <00:10:27.704>anything,

249
00:10:31.005 --> 00:10:32.905
<00:10:31.005>uh, <00:10:31.160>it <00:10:31.316>would <00:10:31.471>be <00:10:31.627>pretty <00:10:31.782>chaotic <00:10:31.938>if <00:10:32.093>we <00:10:32.249>did <00:10:32.405>that.

250
00:10:33.375 --> 00:10:35.465
<00:10:33.375>Well, <00:10:33.530>we <00:10:33.686>wanna <00:10:33.841>do <00:10:33.997>the <00:10:34.152>same <00:10:34.308>thing <00:10:34.463>for <00:10:34.619>lms, <00:10:34.775>right?

251
00:10:35.925 --> 00:10:38.065
<00:10:35.925>So <00:10:36.099>the <00:10:36.275>idea <00:10:36.449>here <00:10:36.625>is <00:10:36.800>to <00:10:36.974>analyze <00:10:37.150>the <00:10:37.324>inputs

252
00:10:38.065 --> 00:10:39.785
<00:10:38.065>and <00:10:38.298>the <00:10:38.531>outputs <00:10:38.765>just <00:10:38.998>to <00:10:39.231>make <00:10:39.465>sure

253
00:10:39.785 --> 00:10:40.905
<00:10:39.785>that <00:10:39.986>what <00:10:40.188>we're <00:10:40.389>doing <00:10:40.591>makes <00:10:40.793>sense.

254
00:10:42.365 --> 00:10:43.705
<00:10:42.365>For <00:10:42.666>example, <00:10:42.968>uh, <00:10:43.269>in <00:10:43.571>Canada

255
00:10:43.925 --> 00:10:47.225
<00:10:43.925>or <00:10:44.080>Quebec, <00:10:44.236>where <00:10:44.391>I'm <00:10:44.547>from, <00:10:44.702>we <00:10:44.858>use <00:10:45.013>French <00:10:45.169>and <00:10:45.324>English.

256
00:10:47.325 --> 00:10:49.065
<00:10:47.325>You <00:10:47.558>probably <00:10:47.791>have <00:10:48.025>the <00:10:48.258>same <00:10:48.491>issue <00:10:48.725>here

257
00:10:49.065 --> 00:10:50.105
<00:10:49.065>with <00:10:49.533>multiple <00:10:50.001>languages.

258
00:10:50.485 --> 00:10:53.785
<00:10:50.485>Um, <00:10:50.640>but <00:10:50.796>if <00:10:50.951>your <00:10:51.107>data <00:10:51.262>set <00:10:51.418>is <00:10:51.573>all <00:10:51.729>in <00:10:51.885>English

259
00:10:53.885 --> 00:10:55.265
<00:10:53.885>and <00:10:54.092>you're <00:10:54.299>asking <00:10:54.506>a <00:10:54.713>question <00:10:54.920>in <00:10:55.127>French,

260
00:10:55.455 --> 00:10:57.345
<00:10:55.455>sometimes <00:10:55.655>the <00:10:55.855>answer <00:10:56.055>will <00:10:56.255>come <00:10:56.455>out <00:10:56.655>in <00:10:56.855>English.

261
00:10:57.965 --> 00:11:01.225
<00:10:57.965>So, <00:10:58.140>one, <00:10:58.315>uh, <00:10:58.490>reflection <00:10:58.665>step <00:10:58.840>could <00:10:59.015>be <00:10:59.190>to <00:10:59.365>validate

262
00:11:01.225 --> 00:11:03.345
<00:11:01.225>that <00:11:01.365>your <00:11:01.505>input <00:11:01.645>language <00:11:01.785>is <00:11:01.925>the <00:11:02.065>same <00:11:02.205>as <00:11:02.345>the <00:11:02.485>output <00:11:02.625>language.

263
00:11:05.445 --> 00:11:06.625
<00:11:05.445>So <00:11:05.799>a <00:11:06.153>popular <00:11:06.507>method

264
00:11:06.765 --> 00:11:08.505
<00:11:06.765>for <00:11:07.045>reflection <00:11:07.324>is <00:11:07.605>through <00:11:07.885>corrective <00:11:08.165>rag.

265
00:11:09.165 --> 00:11:11.545
<00:11:09.165>Uh, <00:11:09.365>here <00:11:09.564>we <00:11:09.765>have <00:11:09.964>exactly <00:11:10.165>the <00:11:10.365>same <00:11:10.564>pipeline

266
00:11:11.545 --> 00:11:14.265
<00:11:11.545>that <00:11:11.719>we <00:11:11.895>usually <00:11:12.069>do, <00:11:12.245>but <00:11:12.420>instead, <00:11:12.594>when <00:11:12.770>we <00:11:12.944>get

267
00:11:14.265 --> 00:11:17.025
<00:11:14.265>to <00:11:14.420>our <00:11:14.576>Vector <00:11:14.731>database, <00:11:14.887>um, <00:11:15.042>uh, <00:11:15.198>we <00:11:15.353>get <00:11:15.509>our <00:11:15.665>results.

268
00:11:17.325 --> 00:11:20.505
<00:11:17.325>We <00:11:17.500>go <00:11:17.675>through <00:11:17.850>a <00:11:18.025>grading <00:11:18.200>phase <00:11:18.375>here, <00:11:18.550>we <00:11:18.725>want

269
00:11:20.505 --> 00:11:23.145
<00:11:20.505>to <00:11:20.660>include <00:11:20.816>an <00:11:20.971>evaluator <00:11:21.127>to <00:11:21.282>make <00:11:21.438>sure <00:11:21.593>that <00:11:21.749>the <00:11:21.905>documents

270
00:11:23.145 --> 00:11:25.785
<00:11:23.145>that <00:11:23.319>we <00:11:23.495>retrieve <00:11:23.670>are <00:11:23.845>actually <00:11:24.020>relevant <00:11:24.194>to <00:11:24.370>our <00:11:24.545>query.

271
00:11:26.525 --> 00:11:27.785
<00:11:26.525>You <00:11:26.714>can <00:11:26.903>do <00:11:27.092>this <00:11:27.281>in <00:11:27.470>multiple <00:11:27.659>ways.

272
00:11:28.125 --> 00:11:30.665
<00:11:28.125>You <00:11:28.280>can <00:11:28.436>use, <00:11:28.591>you <00:11:28.747>know, <00:11:28.902>the <00:11:29.058>same <00:11:29.213>LLM <00:11:29.369>you're <00:11:29.525>using

273
00:11:30.665 --> 00:11:35.105
<00:11:30.665>to <00:11:30.820>generate <00:11:30.976>your <00:11:31.131>response <00:11:31.287>to <00:11:31.442>grade, <00:11:31.598>uh, <00:11:31.753>the <00:11:31.909>the <00:11:32.064>results.

274
00:11:35.445 --> 00:11:37.745
<00:11:35.445>Uh, <00:11:35.572>what <00:11:35.699>you <00:11:35.826>can <00:11:35.954>do <00:11:36.081>is <00:11:36.208>also <00:11:36.335>use <00:11:36.463>a <00:11:36.590>more <00:11:36.717>lightweight <00:11:36.845>model

275
00:11:38.215 --> 00:11:41.715
<00:11:38.215>that, <00:11:38.415>um, <00:11:38.615>that <00:11:38.815>is <00:11:39.015>specific <00:11:39.215>to <00:11:39.415>grading <00:11:39.615>results.

276
00:11:41.735 --> 00:11:42.995
<00:11:41.735>You <00:11:42.018>can <00:11:42.302>use <00:11:42.585>cross <00:11:42.869>encoders,

277
00:11:43.335 --> 00:11:45.395
<00:11:43.335>but <00:11:43.535>ultimately <00:11:43.735>what <00:11:43.935>we <00:11:44.135>want <00:11:44.335>is <00:11:44.535>just <00:11:44.735>to

278
00:11:46.155 --> 00:11:47.795
<00:11:46.155>separate <00:11:46.505>the <00:11:46.855>results <00:11:47.204>into <00:11:47.555>irrelevant

279
00:11:47.795 --> 00:11:48.835
<00:11:47.795>ones <00:11:48.107>and <00:11:48.419>correct <00:11:48.731>results.

280
00:11:49.585 --> 00:11:50.715
<00:11:49.585>When <00:11:49.839>we <00:11:50.093>have <00:11:50.347>correct <00:11:50.602>results,

281
00:11:50.775 --> 00:11:52.195
<00:11:50.775>we <00:11:50.988>can <00:11:51.201>go <00:11:51.414>through <00:11:51.627>as <00:11:51.840>refinement <00:11:52.053>steps.

282
00:11:52.195 --> 00:11:55.205
<00:11:52.195>Since <00:11:52.335>we <00:11:52.475>graded <00:11:52.615>the <00:11:52.755>the <00:11:52.895>results, <00:11:53.035>we <00:11:53.175>can <00:11:53.315>also <00:11:53.455>look <00:11:53.595>at

283
00:11:55.395 --> 00:11:58.125
<00:11:55.395>what <00:11:55.569>information's <00:11:55.745>important, <00:11:55.920>uh, <00:11:56.095>what <00:11:56.270>was <00:11:56.444>found <00:11:56.620>that <00:11:56.795>was,

284
00:11:58.125 --> 00:11:59.365
<00:11:58.125>that <00:11:58.348>was <00:11:58.571>pertinent <00:11:58.794>to <00:11:59.017>our <00:11:59.241>query.

285
00:12:00.105 --> 00:12:01.485
<00:12:00.105>And <00:12:00.415>taking <00:12:00.726>these <00:12:01.036>knowledge <00:12:01.347>trips,

286
00:12:01.545 --> 00:12:03.485
<00:12:01.545>we <00:12:01.719>can <00:12:01.895>condense <00:12:02.069>them <00:12:02.245>into <00:12:02.420>one <00:12:02.594>block <00:12:02.770>of <00:12:02.944>text

287
00:12:03.515 --> 00:12:05.285
<00:12:03.515>that <00:12:03.715>we <00:12:03.915>can <00:12:04.115>then <00:12:04.314>send <00:12:04.515>to <00:12:04.715>an <00:12:04.915>LLM

288
00:12:05.945 --> 00:12:07.765
<00:12:05.945>for <00:12:06.145>less <00:12:06.345>noise <00:12:06.545>in <00:12:06.745>our, <00:12:06.945>in <00:12:07.145>our <00:12:07.345>context.

289
00:12:09.545 --> 00:12:12.045
<00:12:09.545>So <00:12:09.719>on <00:12:09.895>the <00:12:10.069>other <00:12:10.245>side, <00:12:10.420>we <00:12:10.594>can <00:12:10.770>do, <00:12:10.944>uh,

290
00:12:12.055 --> 00:12:13.205
<00:12:12.055>irrelevant <00:12:12.572>results, <00:12:13.090>right?

291
00:12:13.205 --> 00:12:15.365
<00:12:13.205>We <00:12:13.380>can <00:12:13.555>take <00:12:13.730>corrective <00:12:13.905>measures <00:12:14.080>if <00:12:14.255>we <00:12:14.430>figure <00:12:14.605>out

292
00:12:15.365 --> 00:12:17.205
<00:12:15.365>that <00:12:15.645>everything <00:12:15.925>that <00:12:16.205>we <00:12:16.485>got <00:12:16.765>has

293
00:12:17.835 --> 00:12:19.285
<00:12:17.835>nothing <00:12:18.096>to <00:12:18.357>do <00:12:18.618>with <00:12:18.879>our <00:12:19.140>query.

294
00:12:20.385 --> 00:12:22.125
<00:12:20.385>So <00:12:20.540>what <00:12:20.696>do <00:12:20.851>we <00:12:21.007>do <00:12:21.162>here? <00:12:21.318>Well, <00:12:21.473>we <00:12:21.629>can <00:12:21.785>default

295
00:12:22.125 --> 00:12:23.125
<00:12:22.125>to <00:12:22.350>a <00:12:22.575>different <00:12:22.800>data <00:12:23.025>source,

296
00:12:24.075 --> 00:12:26.455
<00:12:24.075>or <00:12:24.275>we <00:12:24.475>can <00:12:24.675>do <00:12:24.875>something <00:12:25.075>like <00:12:25.275>query <00:12:25.475>translations.

297
00:12:26.555 --> 00:12:28.255
<00:12:26.555>So <00:12:26.788>when <00:12:27.021>we <00:12:27.255>are <00:12:27.488>doing <00:12:27.721>query <00:12:27.954>translations,

298
00:12:28.255 --> 00:12:30.135
<00:12:28.255>we're <00:12:28.488>rewriting <00:12:28.721>our <00:12:28.955>query <00:12:29.188>in <00:12:29.421>the <00:12:29.655>hopes

299
00:12:30.235 --> 00:12:32.175
<00:12:30.235>of <00:12:30.701>getting <00:12:31.168>different <00:12:31.635>results.

300
00:12:32.485 --> 00:12:37.255
<00:12:32.485>Okay? <00:12:32.640>And <00:12:32.796>this <00:12:32.951>sort <00:12:33.107>of <00:12:33.262>leads <00:12:33.418>us <00:12:33.573>to, <00:12:33.729>um, <00:12:33.885>oh,

301
00:12:37.385 --> 00:12:40.255
<00:12:37.385>sorry, <00:12:37.618>to <00:12:37.851>query <00:12:38.085>translation <00:12:38.318>and <00:12:38.551>rag <00:12:38.785>fusion.

302
00:12:41.195 --> 00:12:43.615
<00:12:41.195>Uh, <00:12:41.370>this <00:12:41.545>is <00:12:41.720>also, <00:12:41.895>I <00:12:42.070>think, <00:12:42.245>called <00:12:42.420>query <00:12:42.595>expansion.

303
00:12:43.745 --> 00:12:46.775
<00:12:43.745>We're <00:12:43.978>not, <00:12:44.211>nobody's, <00:12:44.445>uh, <00:12:44.678>concluded <00:12:44.911>that <00:12:45.145>yet,

304
00:12:47.555 --> 00:12:51.295
<00:12:47.555>but <00:12:47.694>here <00:12:47.834>we're <00:12:47.974>rewriting <00:12:48.114>our <00:12:48.255>query, <00:12:48.395>uh, <00:12:48.535>into <00:12:48.675>an <00:12:48.814>X <00:12:48.954>amount

305
00:12:51.295 --> 00:12:53.015
<00:12:51.295>of <00:12:51.761>different <00:12:52.228>perspectives, <00:12:52.694>okay?

306
00:12:53.015 --> 00:12:54.415
<00:12:53.015>From <00:12:53.266>the <00:12:53.519>eyes <00:12:53.771>of <00:12:54.023>different <00:12:54.275>stakeholders.

307
00:12:55.635 --> 00:12:58.095
<00:12:55.635>And <00:12:55.790>then <00:12:55.946>we're <00:12:56.101>mixing <00:12:56.257>all <00:12:56.412>the <00:12:56.568>results <00:12:56.723>back <00:12:56.879>in <00:12:57.035>together

308
00:12:58.095 --> 00:12:59.575
<00:12:58.095>with <00:12:58.428>reciprocal <00:12:58.761>rank <00:12:59.094>Fusion <00:12:59.427>again.

309
00:12:59.635 --> 00:13:02.935
<00:12:59.635>So, <00:12:59.762>just <00:12:59.889>as <00:13:00.016>an <00:13:00.144>example, <00:13:00.271>this <00:13:00.398>is <00:13:00.525>pulled <00:13:00.653>from <00:13:00.780>the <00:13:00.907>RFP <00:13:01.035>uh,

310
00:13:03.095 --> 00:13:04.135
<00:13:03.095>rag <00:13:03.282>pipeline <00:13:03.469>that <00:13:03.656>I <00:13:03.843>talked <00:13:04.031>about.

311
00:13:04.915 --> 00:13:07.775
<00:13:04.915>What's <00:13:05.194>the <00:13:05.474>average <00:13:05.755>annual <00:13:06.035>energy <00:13:06.314>expenditure,

312
00:13:08.155 --> 00:13:09.215
<00:13:08.155>uh, <00:13:08.473>for <00:13:08.791>hockey <00:13:09.109>rinks?

313
00:13:09.735 --> 00:13:12.135
<00:13:09.735>'cause <00:13:10.435>Canada, <00:13:11.135>uh,

314
00:13:12.195 --> 00:13:15.015
<00:13:12.195>and <00:13:12.350>here <00:13:12.506>we <00:13:12.661>have <00:13:12.817>the <00:13:12.972>query <00:13:13.128>rewritten <00:13:13.283>five <00:13:13.439>different <00:13:13.595>times,

315
00:13:15.195 --> 00:13:16.695
<00:13:15.195>but <00:13:15.645>from <00:13:16.095>different <00:13:16.545>perspectives.

316
00:13:16.695 --> 00:13:20.335
<00:13:16.695>So <00:13:16.850>project <00:13:17.006>manager <00:13:17.161>is <00:13:17.317>adding <00:13:17.472>cost <00:13:17.628>savings <00:13:17.783>into <00:13:17.939>the <00:13:18.095>text.

317
00:13:20.595 --> 00:13:23.135
<00:13:20.595>Uh, <00:13:20.795>financial <00:13:20.995>analyst <00:13:21.195>is <00:13:21.395>looking <00:13:21.595>for <00:13:21.795>cost <00:13:21.995>efficiencies.

318
00:13:23.635 --> 00:13:25.415
<00:13:23.635>And <00:13:23.810>what <00:13:23.985>you're <00:13:24.160>doing <00:13:24.335>here <00:13:24.510>is <00:13:24.685>you're <00:13:24.860>essentially <00:13:25.035>just

319
00:13:25.415 --> 00:13:26.655
<00:13:25.415>widening <00:13:25.786>your <00:13:26.159>net, <00:13:26.531>right?

320
00:13:26.675 --> 00:13:27.935
<00:13:26.675>If <00:13:26.901>you're <00:13:27.128>fishing <00:13:27.355>for <00:13:27.582>an <00:13:27.809>answer,

321
00:13:28.155 --> 00:13:29.575
<00:13:28.155>you're <00:13:28.410>just <00:13:28.666>casting <00:13:28.921>a <00:13:29.177>bigger <00:13:29.433>net

322
00:13:29.595 --> 00:13:31.455
<00:13:29.595>to <00:13:29.795>get <00:13:29.995>to <00:13:30.195>the <00:13:30.395>answer <00:13:30.595>that <00:13:30.795>you <00:13:30.995>want.

323
00:13:33.805 --> 00:13:38.655
<00:13:33.805>Another <00:13:33.979>important <00:13:34.155>aspect <00:13:34.329>of <00:13:34.505>agentic <00:13:34.680>systems <00:13:34.854>is <00:13:35.030>tool <00:13:35.204>use.

324
00:13:38.915 --> 00:13:41.135
<00:13:38.915>And <00:13:39.115>this <00:13:39.314>is <00:13:39.515>really <00:13:39.714>basic, <00:13:39.915>it's <00:13:40.115>function <00:13:40.314>calling.

325
00:13:41.875 --> 00:13:44.855
<00:13:41.875>Uh, <00:13:42.002>so <00:13:42.129>the <00:13:42.256>idea <00:13:42.384>here <00:13:42.511>is <00:13:42.638>to <00:13:42.765>provide <00:13:42.893>a <00:13:43.020>list <00:13:43.147>of <00:13:43.275>actions

326
00:13:44.855 --> 00:13:48.215
<00:13:44.855>that <00:13:45.055>your <00:13:45.255>LLM <00:13:45.455>can <00:13:45.655>take, <00:13:45.855>uh, <00:13:46.055>and <00:13:46.255>descriptions

327
00:13:48.275 --> 00:13:49.655
<00:13:48.275>and <00:13:48.523>parameters <00:13:48.771>that <00:13:49.020>it <00:13:49.268>could <00:13:49.516>use,

328
00:13:50.115 --> 00:13:52.815
<00:13:50.115>and <00:13:50.270>basically <00:13:50.426>let <00:13:50.581>it <00:13:50.737>choose, <00:13:50.892>uh, <00:13:51.048>between <00:13:51.203>all <00:13:51.359>the <00:13:51.515>tools.

329
00:13:52.815 --> 00:13:54.815
<00:13:52.815>So <00:13:52.970>this <00:13:53.126>is <00:13:53.281>a <00:13:53.437>great <00:13:53.592>way <00:13:53.748>to <00:13:53.903>abstract <00:13:54.059>all <00:13:54.215>your,

330
00:13:55.045 --> 00:13:56.175
<00:13:55.045>your <00:13:55.553>core <00:13:56.061>functionalities.

331
00:13:57.565 --> 00:13:59.775
<00:13:57.565>Some <00:13:57.720>examples <00:13:57.876>that <00:13:58.031>you <00:13:58.187>can <00:13:58.342>do, <00:13:58.498>uh, <00:13:58.653>as <00:13:58.809>web <00:13:58.965>search,

332
00:13:59.875 --> 00:14:01.575
<00:13:59.875>vector <00:14:00.341>search, <00:14:00.808>SQL <00:14:01.275>search.

333
00:14:01.595 --> 00:14:03.535
<00:14:01.595>But <00:14:01.795>you <00:14:01.995>can <00:14:02.195>also <00:14:02.395>do <00:14:02.595>other <00:14:02.795>things, <00:14:02.995>right?

334
00:14:03.615 --> 00:14:05.255
<00:14:03.615>You <00:14:03.848>don't <00:14:04.081>have <00:14:04.315>to <00:14:04.548>always <00:14:04.781>do <00:14:05.015>retrieval.

335
00:14:05.255 --> 00:14:08.575
<00:14:05.255>You <00:14:05.362>can <00:14:05.470>do, <00:14:05.578>uh, <00:14:05.685>you <00:14:05.793>can <00:14:05.901>use <00:14:06.008>a <00:14:06.116>calculator, <00:14:06.224>you <00:14:06.331>can <00:14:06.439>go <00:14:06.547>as <00:14:06.655>far

336
00:14:08.575 --> 00:14:10.215
<00:14:08.575>as <00:14:08.925>doing <00:14:09.275>computer <00:14:09.625>vision <00:14:09.975>workloads.

337
00:14:11.165 --> 00:14:14.705
<00:14:11.165>Uh, <00:14:11.292>and <00:14:11.419>another <00:14:11.546>cool <00:14:11.674>tool <00:14:11.801>that <00:14:11.928>you <00:14:12.055>can <00:14:12.183>do <00:14:12.310>is <00:14:12.437>a <00:14:12.564>tool

338
00:14:14.705 --> 00:14:15.905
<00:14:14.705>to <00:14:15.065>generate <00:14:15.425>tools, <00:14:15.785>right?

339
00:14:16.405 --> 00:14:19.985
<00:14:16.405>So <00:14:16.579>now <00:14:16.755>we're <00:14:16.930>including <00:14:17.105>this <00:14:17.280>like <00:14:17.454>self-improving <00:14:17.630>loop, <00:14:17.805>um,

340
00:14:20.575 --> 00:14:21.785
<00:14:20.575>into <00:14:20.938>our, <00:14:21.301>our <00:14:21.664>agent.

341
00:14:22.045 --> 00:14:26.065
<00:14:22.045>And <00:14:22.200>it <00:14:22.356>can <00:14:22.511>do <00:14:22.667>some <00:14:22.822>amazing <00:14:22.978>things <00:14:23.133>less, <00:14:23.289>uh, <00:14:23.444>pertinent

342
00:14:26.065 --> 00:14:28.785
<00:14:26.065>to <00:14:26.192>rag, <00:14:26.319>but <00:14:26.446>I <00:14:26.574>think <00:14:26.701>it's <00:14:26.828>really <00:14:26.955>cool <00:14:27.083>to <00:14:27.210>at <00:14:27.337>least <00:14:27.465>mention

343
00:14:28.815 --> 00:14:30.865
<00:14:28.815>that <00:14:29.015>these <00:14:29.215>LMS <00:14:29.415>can <00:14:29.615>build <00:14:29.815>their <00:14:30.015>own <00:14:30.215>tools.

344
00:14:33.145 --> 00:14:36.245
<00:14:33.145>So <00:14:33.378>another <00:14:33.611>probably <00:14:33.845>the <00:14:34.078>most <00:14:34.311>important <00:14:34.545>aspect

345
00:14:36.345 --> 00:14:38.845
<00:14:36.345>of <00:14:36.520>these <00:14:36.695>agentic <00:14:36.870>systems <00:14:37.045>is <00:14:37.220>the <00:14:37.395>planning <00:14:37.570>phase, <00:14:37.745>okay?

346
00:14:39.005 --> 00:14:41.365
<00:14:39.005>'cause <00:14:39.145>this <00:14:39.285>is <00:14:39.425>where <00:14:39.564>we <00:14:39.705>can <00:14:39.845>start <00:14:39.985>working <00:14:40.125>towards <00:14:40.265>our <00:14:40.405>goal

347
00:14:41.665 --> 00:14:44.085
<00:14:41.665>and <00:14:41.865>breaking <00:14:42.064>down <00:14:42.265>these <00:14:42.464>queries <00:14:42.665>into <00:14:42.865>sub <00:14:43.064>queries

348
00:14:44.385 --> 00:14:45.805
<00:14:44.385>and <00:14:44.640>giving <00:14:44.896>a <00:14:45.151>tool <00:14:45.407>each <00:14:45.663>time.

349
00:14:47.505 --> 00:14:50.525
<00:14:47.505>The <00:14:47.680>most <00:14:47.855>basic <00:14:48.030>form <00:14:48.205>of <00:14:48.380>planning <00:14:48.555>is <00:14:48.730>through <00:14:48.905>routing.

350
00:14:51.185 --> 00:14:53.205
<00:14:51.185>And <00:14:51.340>when <00:14:51.496>you're <00:14:51.651>routing, <00:14:51.807>all <00:14:51.962>you're <00:14:52.118>doing <00:14:52.273>is <00:14:52.429>saying, <00:14:52.584>Hey,

351
00:14:53.305 --> 00:14:56.765
<00:14:53.305>out <00:14:53.444>of <00:14:53.584>all <00:14:53.724>these <00:14:53.864>tools, <00:14:54.005>uh, <00:14:54.145>just <00:14:54.285>based <00:14:54.425>on <00:14:54.564>this <00:14:54.704>query,

352
00:14:57.075 --> 00:14:59.125
<00:14:57.075>give <00:14:57.250>me <00:14:57.425>one <00:14:57.600>tool <00:14:57.775>that <00:14:57.950>might <00:14:58.125>answer <00:14:58.300>my <00:14:58.475>question.

353
00:15:00.025 --> 00:15:03.085
<00:15:00.025>And <00:15:00.165>this <00:15:00.305>can <00:15:00.444>be <00:15:00.584>the <00:15:00.725>decision <00:15:00.865>between <00:15:01.005>a, <00:15:01.145>a <00:15:01.285>vector <00:15:01.425>search

354
00:15:03.085 --> 00:15:06.405
<00:15:03.085>or <00:15:03.285>an <00:15:03.485>SQL <00:15:03.685>search, <00:15:03.885>Going <00:15:04.085>a <00:15:04.285>step <00:15:04.485>beyond,

355
00:15:06.505 --> 00:15:08.045
<00:15:06.505>we <00:15:06.782>can <00:15:07.059>do <00:15:07.336>query <00:15:07.613>planning, <00:15:07.891>okay?

356
00:15:08.225 --> 00:15:10.325
<00:15:08.225>So <00:15:08.400>when <00:15:08.575>we're <00:15:08.750>doing <00:15:08.925>query <00:15:09.100>planning, <00:15:09.275>we're <00:15:09.450>saying, <00:15:09.625>Hey,

357
00:15:10.335 --> 00:15:13.605
<00:15:10.335>based <00:15:10.490>off <00:15:10.646>this <00:15:10.801>query, <00:15:10.957>break <00:15:11.112>it <00:15:11.268>down <00:15:11.423>into <00:15:11.579>multiple <00:15:11.735>steps

358
00:15:13.605 --> 00:15:15.445
<00:15:13.605>and <00:15:13.805>give <00:15:14.005>me <00:15:14.205>an <00:15:14.405>action <00:15:14.605>for <00:15:14.805>each <00:15:15.005>step.

359
00:15:15.585 --> 00:15:17.845
<00:15:15.585>And <00:15:15.740>then <00:15:15.896>at <00:15:16.051>the <00:15:16.207>end, <00:15:16.362>we'll <00:15:16.518>synthesize <00:15:16.673>the <00:15:16.829>results <00:15:16.985>together,

360
00:15:18.155 --> 00:15:20.485
<00:15:18.155>basically <00:15:18.505>work <00:15:18.855>towards <00:15:19.204>our <00:15:19.555>answer

361
00:15:25.225 --> 00:15:27.945
<00:15:25.225>a <00:15:25.380>much <00:15:25.536>more, <00:15:25.691>uh, <00:15:25.847>intense <00:15:26.002>approach <00:15:26.158>is <00:15:26.313>the <00:15:26.469>react <00:15:26.625>loop,

362
00:15:28.005 --> 00:15:30.785
<00:15:28.005>and <00:15:28.180>it's <00:15:28.355>not <00:15:28.530>the, <00:15:28.705>the <00:15:28.880>JavaScript <00:15:29.055>framework, <00:15:29.230>it's <00:15:29.405>actually

363
00:15:30.785 --> 00:15:33.545
<00:15:30.785>what <00:15:30.912>auto <00:15:31.039>GPT <00:15:31.166>used, <00:15:31.294>uh, <00:15:31.421>when <00:15:31.548>it, <00:15:31.675>when <00:15:31.803>it <00:15:31.930>first <00:15:32.057>came <00:15:32.185>out.

364
00:15:33.585 --> 00:15:34.625
<00:15:33.585>I <00:15:33.718>don't <00:15:33.852>know <00:15:33.986>if <00:15:34.119>it's <00:15:34.253>still <00:15:34.387>the <00:15:34.521>case.

365
00:15:35.245 --> 00:15:37.225
<00:15:35.245>Um, <00:15:35.525>but <00:15:35.805>in <00:15:36.085>this <00:15:36.365>scenario, <00:15:36.645>this,

366
00:15:37.235 --> 00:15:38.825
<00:15:37.235>we're <00:15:37.585>doing <00:15:37.935>something <00:15:38.285>interesting <00:15:38.635>here.

367
00:15:39.525 --> 00:15:41.985
<00:15:39.525>Uh, <00:15:39.725>we're <00:15:39.925>giving <00:15:40.125>a <00:15:40.324>thought <00:15:40.525>for <00:15:40.725>the <00:15:40.925>query

368
00:15:41.985 --> 00:15:43.825
<00:15:41.985>that <00:15:42.140>we <00:15:42.296>just <00:15:42.451>received, <00:15:42.607>and <00:15:42.762>then <00:15:42.918>we're <00:15:43.073>getting <00:15:43.229>an <00:15:43.385>action.

369
00:15:44.525 --> 00:15:47.065
<00:15:44.525>And <00:15:44.805>then <00:15:45.084>after <00:15:45.365>executing <00:15:45.645>this <00:15:45.925>action,

370
00:15:47.635 --> 00:15:49.385
<00:15:47.635>we're <00:15:47.868>giving <00:15:48.101>an <00:15:48.335>observation <00:15:48.568>on <00:15:48.801>the <00:15:49.035>action,

371
00:15:49.805 --> 00:15:52.225
<00:15:49.805>and <00:15:49.944>we're <00:15:50.084>running <00:15:50.224>this <00:15:50.364>in <00:15:50.505>a <00:15:50.645>loop <00:15:50.785>until <00:15:50.925>we <00:15:51.064>actually <00:15:51.204>get

372
00:15:52.225 --> 00:15:53.265
<00:15:52.225>to <00:15:52.412>the <00:15:52.599>result <00:15:52.786>that <00:15:52.973>we <00:15:53.161>want.

373
00:15:54.565 --> 00:15:57.025
<00:15:54.565>So <00:15:54.740>in <00:15:54.915>this <00:15:55.090>scenario <00:15:55.265>here, <00:15:55.440>we're <00:15:55.615>looking <00:15:55.790>at, <00:15:55.965>um,

374
00:15:57.675 --> 00:16:00.905
<00:15:57.675>we're <00:15:57.814>looking <00:15:57.954>at <00:15:58.094>a <00:15:58.234>vector <00:15:58.375>query <00:15:58.515>engine <00:15:58.655>to <00:15:58.795>get <00:15:58.935>the <00:15:59.074>IDs

375
00:16:00.905 --> 00:16:03.025
<00:16:00.905>of <00:16:01.079>the <00:16:01.255>hockey <00:16:01.430>rinks <00:16:01.605>in <00:16:01.780>our, <00:16:01.954>in <00:16:02.130>our <00:16:02.305>dataset.

376
00:16:03.335 --> 00:16:06.815
<00:16:03.335>Then <00:16:03.475>after <00:16:03.615>that, <00:16:03.755>we're <00:16:03.895>moving <00:16:04.035>to <00:16:04.175>step <00:16:04.315>two, <00:16:04.455>where <00:16:04.595>we're <00:16:04.735>going

377
00:16:06.815 --> 00:16:08.575
<00:16:06.815>through <00:16:07.015>an <00:16:07.215>SQL <00:16:07.415>database <00:16:07.615>to <00:16:07.815>get <00:16:08.015>the <00:16:08.215>results.

378
00:16:08.915 --> 00:16:10.415
<00:16:08.915>And <00:16:09.140>then <00:16:09.365>finally, <00:16:09.589>we're <00:16:09.814>getting <00:16:10.040>the <00:16:10.265>finished

379
00:16:10.415 --> 00:16:11.815
<00:16:10.415>keyword <00:16:10.730>to <00:16:11.045>end <00:16:11.360>the <00:16:11.675>loop.

380
00:16:13.115 --> 00:16:15.885
<00:16:13.115>There's <00:16:13.270>A <00:16:13.426>lot <00:16:13.581>of <00:16:13.737>issues <00:16:13.892>with <00:16:14.048>this <00:16:14.203>one. <00:16:14.359>Infinite <00:16:14.515>loops.

381
00:16:16.265 --> 00:16:18.605
<00:16:16.265>If <00:16:16.392>you <00:16:16.519>can't <00:16:16.646>get <00:16:16.774>to <00:16:16.901>your <00:16:17.028>answer, <00:16:17.155>you <00:16:17.283>might <00:16:17.410>just <00:16:17.537>get <00:16:17.665>stuck

382
00:16:18.705 --> 00:16:19.925
<00:16:18.705>and <00:16:18.842>it's <00:16:18.979>gonna <00:16:19.116>cost <00:16:19.254>you <00:16:19.391>a <00:16:19.528>lot <00:16:19.665>of <00:16:19.803>money.

383
00:16:20.585 --> 00:16:22.845
<00:16:20.585>Uh, <00:16:20.760>and <00:16:20.935>another <00:16:21.110>one <00:16:21.285>is <00:16:21.460>it's <00:16:21.635>very <00:16:21.810>sequential, <00:16:21.985>right?

384
00:16:23.185 --> 00:16:25.085
<00:16:23.185>You <00:16:23.464>get <00:16:23.744>sort <00:16:24.025>of <00:16:24.305>stuck, <00:16:24.584>uh,

385
00:16:25.905 --> 00:16:27.845
<00:16:25.905>for <00:16:26.105>a <00:16:26.305>long <00:16:26.505>time <00:16:26.704>working <00:16:26.905>towards <00:16:27.105>an <00:16:27.305>answer.

386
00:16:29.345 --> 00:16:32.925
<00:16:29.345>So <00:16:29.545>going <00:16:29.745>beyond <00:16:29.945>this, <00:16:30.145>we <00:16:30.345>have <00:16:30.545>these, <00:16:30.745>uh,

387
00:16:32.925 --> 00:16:34.725
<00:16:32.925>these <00:16:33.391>LLM <00:16:33.858>orchestration <00:16:34.324>frameworks.

388
00:16:34.725 --> 00:16:36.685
<00:16:34.725>They're <00:16:35.005>adopting <00:16:35.285>a <00:16:35.565>DAG <00:16:35.845>based <00:16:36.125>approach,

389
00:16:36.745 --> 00:16:38.445
<00:16:36.745>so <00:16:37.095>directed <00:16:37.445>a <00:16:37.795>cyclic <00:16:38.145>graphs.

390
00:16:38.635 --> 00:16:41.965
<00:16:38.635>They're <00:16:38.835>your, <00:16:39.035>your <00:16:39.235>standard, <00:16:39.435>um, <00:16:39.635>your <00:16:39.835>standard <00:16:40.035>graphs

391
00:16:41.985 --> 00:16:44.125
<00:16:41.985>for <00:16:42.218>like <00:16:42.451>all <00:16:42.685>your <00:16:42.918>data <00:16:43.151>orchestration <00:16:43.385>pipelines.

392
00:16:44.385 --> 00:16:47.245
<00:16:44.385>Uh, <00:16:44.512>and <00:16:44.639>what <00:16:44.766>we're <00:16:44.894>gonna <00:16:45.021>do <00:16:45.148>here <00:16:45.275>is <00:16:45.403>we're <00:16:45.530>gonna <00:16:45.657>model <00:16:45.785>the

393
00:16:47.245 --> 00:16:49.805
<00:16:47.245>dependencies <00:16:47.525>and <00:16:47.805>the <00:16:48.085>relationships <00:16:48.365>between <00:16:48.645>all

394
00:16:49.805 --> 00:16:53.365
<00:16:49.805>of <00:16:49.944>our <00:16:50.084>components <00:16:50.224>in <00:16:50.364>the <00:16:50.505>rack <00:16:50.645>process <00:16:50.785>using <00:16:50.925>edges <00:16:51.064>and <00:16:51.204>nodes.

395
00:16:53.825 --> 00:16:56.245
<00:16:53.825>So <00:16:54.025>this <00:16:54.225>is <00:16:54.425>very <00:16:54.625>declarative <00:16:54.825>and <00:16:55.025>it's <00:16:55.225>observable,

396
00:16:56.625 --> 00:16:58.005
<00:16:56.625>and <00:16:56.935>it <00:16:57.246>allows <00:16:57.556>us <00:16:57.867>to

397
00:16:58.625 --> 00:17:01.125
<00:16:58.625>really <00:16:58.825>debug <00:16:59.025>what's <00:16:59.225>going <00:16:59.425>on <00:16:59.625>in <00:16:59.825>our <00:17:00.025>pipeline.

398
00:17:02.385 --> 00:17:05.525
<00:17:02.385>And <00:17:02.512>using <00:17:02.639>these <00:17:02.766>dags, <00:17:02.894>we <00:17:03.021>can <00:17:03.148>go <00:17:03.275>to <00:17:03.403>a <00:17:03.530>more <00:17:03.657>advanced <00:17:03.785>approach.

399
00:17:05.705 --> 00:17:07.845
<00:17:05.704>Uh, <00:17:05.905>the <00:17:06.105>LLM <00:17:06.304>compiler <00:17:06.504>is <00:17:06.704>an <00:17:06.905>interesting <00:17:07.105>project.

400
00:17:09.385 --> 00:17:10.805
<00:17:09.385>So <00:17:09.598>here <00:17:09.810>we <00:17:10.024>have <00:17:10.237>a <00:17:10.450>planning <00:17:10.663>phase,

401
00:17:11.345 --> 00:17:12.485
<00:17:11.345>and <00:17:11.601>during <00:17:11.858>the <00:17:12.114>planning <00:17:12.370>phase,

402
00:17:12.485 --> 00:17:14.285
<00:17:12.484>we're <00:17:12.685>gonna <00:17:12.885>generate <00:17:13.084>our <00:17:13.284>own <00:17:13.484>dag <00:17:13.685>of <00:17:13.885>tasks.

403
00:17:14.665 --> 00:17:18.245
<00:17:14.665>And <00:17:14.820>using <00:17:14.976>these <00:17:15.131>dags, <00:17:15.287>we <00:17:15.442>can <00:17:15.598>also <00:17:15.753>identify <00:17:15.909>the <00:17:16.065>dependencies.

404
00:17:19.185 --> 00:17:23.165
<00:17:19.185>And <00:17:19.359>by <00:17:19.534>reordering <00:17:19.710>these <00:17:19.885>dependencies, <00:17:20.060>we <00:17:20.234>can <00:17:20.409>create, <00:17:20.585>uh,

405
00:17:23.185 --> 00:17:25.685
<00:17:23.185>an <00:17:23.464>executor, <00:17:23.744>an <00:17:24.024>executor <00:17:24.304>pool, <00:17:24.585>sorry,

406
00:17:25.795 --> 00:17:27.565
<00:17:25.795>that'll <00:17:26.075>be <00:17:26.355>pulling <00:17:26.635>all <00:17:26.915>these <00:17:27.195>tasks

407
00:17:27.825 --> 00:17:29.125
<00:17:27.825>and <00:17:28.117>running <00:17:28.410>them <00:17:28.702>in <00:17:28.995>parallel.

408
00:17:29.905 --> 00:17:32.605
<00:17:29.905>So <00:17:30.105>here <00:17:30.305>we're, <00:17:30.504>uh, <00:17:30.705>we're <00:17:30.905>no <00:17:31.105>longer <00:17:31.305>sequential.

409
00:17:32.945 --> 00:17:35.565
<00:17:32.945>We <00:17:33.145>have, <00:17:33.345>uh, <00:17:33.544>parallel <00:17:33.745>processing <00:17:33.945>for <00:17:34.145>our <00:17:34.345>agent,

410
00:17:35.705 --> 00:17:36.885
<00:17:35.705>uh, <00:17:36.235>rag <00:17:36.767>pipeline.

411
00:17:40.525 --> 00:17:42.965
<00:17:40.525>Finally, <00:17:40.875>another <00:17:41.225>important <00:17:41.575>aspect <00:17:41.925>is

412
00:17:42.965 --> 00:17:44.085
<00:17:42.965>multi-agent <00:17:43.973>collaboration.

413
00:17:44.145 --> 00:17:46.085
<00:17:44.145>And <00:17:44.378>this <00:17:44.611>is <00:17:44.845>more <00:17:45.078>experimental <00:17:45.311>than <00:17:45.545>anything.

414
00:17:47.305 --> 00:17:50.805
<00:17:47.305>So <00:17:47.432>the <00:17:47.559>idea <00:17:47.686>here <00:17:47.814>is <00:17:47.941>to <00:17:48.068>create <00:17:48.195>a <00:17:48.323>proper <00:17:48.450>separation <00:17:48.577>of <00:17:48.705>concern

415
00:17:50.825 --> 00:17:53.765
<00:17:50.825>and <00:17:50.980>not <00:17:51.136>overloading <00:17:51.291>your <00:17:51.447>agent <00:17:51.602>with <00:17:51.758>too <00:17:51.913>much <00:17:52.069>work, <00:17:52.225>right?

416
00:17:54.305 --> 00:17:57.485
<00:17:54.305>So <00:17:54.445>we <00:17:54.585>want <00:17:54.725>to <00:17:54.865>create <00:17:55.005>these <00:17:55.145>atomic <00:17:55.285>agents <00:17:55.425>that <00:17:55.565>can <00:17:55.705>interact

417
00:17:57.485 --> 00:17:58.925
<00:17:57.485>with <00:17:57.701>each <00:17:57.917>other <00:17:58.132>and <00:17:58.349>delegate <00:17:58.565>some <00:17:58.781>work,

418
00:17:59.185 --> 00:18:01.005
<00:17:59.185>but <00:17:59.385>they <00:17:59.585>can <00:17:59.784>also <00:17:59.985>do <00:18:00.185>their <00:18:00.385>own <00:18:00.585>planning.

419
00:18:03.385 --> 00:18:04.725
<00:18:03.385>So <00:18:03.686>an <00:18:03.988>example <00:18:04.289>here, <00:18:04.591>uh,

420
00:18:04.745 --> 00:18:07.765
<00:18:04.745>is <00:18:04.978>the <00:18:05.211>hierarchical <00:18:05.445>multi-agent <00:18:05.678>collaboration, <00:18:05.911>again, <00:18:06.145>for,

421
00:18:08.305 --> 00:18:11.525
<00:18:08.305>um, <00:18:08.480>a <00:18:08.655>rag <00:18:08.830>agent <00:18:09.005>to <00:18:09.180>create <00:18:09.355>requests <00:18:09.530>for <00:18:09.705>proposals.

422
00:18:12.025 --> 00:18:15.525
<00:18:12.025>So <00:18:12.165>we <00:18:12.305>have <00:18:12.445>the <00:18:12.585>proposal <00:18:12.725>manager <00:18:12.865>that <00:18:13.005>has <00:18:13.145>access <00:18:13.285>to <00:18:13.425>like,

423
00:18:15.525 --> 00:18:17.645
<00:18:15.525>maybe <00:18:15.725>a <00:18:15.925>proposal <00:18:16.125>templates <00:18:16.325>and <00:18:16.525>a <00:18:16.725>vector <00:18:16.925>database,

424
00:18:17.945 --> 00:18:21.645
<00:18:17.945>but <00:18:18.120>it <00:18:18.294>also <00:18:18.470>uses <00:18:18.645>delegation <00:18:18.820>as <00:18:18.995>a <00:18:19.169>tool, <00:18:19.345>right?

425
00:18:21.645 --> 00:18:24.325
<00:18:21.645>And <00:18:21.785>it <00:18:21.925>could <00:18:22.065>delegate <00:18:22.205>some <00:18:22.345>of <00:18:22.485>the <00:18:22.625>work <00:18:22.764>to <00:18:22.905>other <00:18:23.045>agents.

426
00:18:25.185 --> 00:18:28.485
<00:18:25.185>So <00:18:25.360>if <00:18:25.534>the <00:18:25.710>question <00:18:25.885>was <00:18:26.060>about, <00:18:26.235>uh, <00:18:26.409>legal <00:18:26.585>assistance,

427
00:18:28.485 --> 00:18:30.285
<00:18:28.485>then <00:18:28.659>maybe <00:18:28.834>it <00:18:29.010>could <00:18:29.185>delegate <00:18:29.360>some <00:18:29.534>of <00:18:29.709>the <00:18:29.885>work

428
00:18:30.285 --> 00:18:32.325
<00:18:30.285>to <00:18:30.485>an <00:18:30.685>illegal <00:18:30.885>agent, <00:18:31.085>and <00:18:31.285>it <00:18:31.485>can <00:18:31.685>answer

429
00:18:32.325 --> 00:18:33.565
<00:18:32.325>using <00:18:32.883>proposal <00:18:33.441>templates.

430
00:18:34.585 --> 00:18:37.325
<00:18:34.585>So <00:18:34.712>you <00:18:34.839>can <00:18:34.966>kind <00:18:35.094>of <00:18:35.221>see <00:18:35.348>how <00:18:35.475>all <00:18:35.603>these <00:18:35.730>pieces <00:18:35.857>start <00:18:35.985>working

431
00:18:37.565 --> 00:18:41.085
<00:18:37.565>together, <00:18:37.740>because <00:18:37.915>all <00:18:38.090>of <00:18:38.265>these <00:18:38.440>agents <00:18:38.615>have <00:18:38.790>their <00:18:38.965>own,

432
00:18:41.745 --> 00:18:43.725
<00:18:41.745>uh, <00:18:42.211>planning <00:18:42.678>method, <00:18:43.145>right?

433
00:18:44.345 --> 00:18:46.965
<00:18:44.345>And <00:18:44.500>they <00:18:44.656>can <00:18:44.811>all <00:18:44.967>work <00:18:45.122>towards <00:18:45.278>an <00:18:45.433>answer <00:18:45.589>in <00:18:45.745>unison.

434
00:18:48.785 --> 00:18:51.085
<00:18:48.785>That's <00:18:48.985>actually <00:18:49.185>where <00:18:49.385>we're <00:18:49.585>gonna <00:18:49.785>stop <00:18:49.985>our <00:18:50.185>exploration

435
00:18:51.085 --> 00:18:52.125
<00:18:51.085>of <00:18:51.553>advanced <00:18:52.021>patterns.

436
00:18:52.425 --> 00:18:53.925
<00:18:52.425>Uh, <00:18:52.649>I <00:18:52.875>think <00:18:53.100>there's <00:18:53.325>more <00:18:53.550>to <00:18:53.774>it,

437
00:18:54.025 --> 00:18:56.965
<00:18:54.025>but <00:18:54.200>at <00:18:54.375>this <00:18:54.550>point <00:18:54.725>it's <00:18:54.900>more <00:18:55.075>experimental <00:18:55.250>than <00:18:55.425>anything.

438
00:18:57.265 --> 00:19:00.645
<00:18:57.265>Uh, <00:18:57.381>but <00:18:57.498>before, <00:18:57.615>I <00:18:57.731>wanna <00:18:57.848>leave <00:18:57.965>with <00:18:58.081>a <00:18:58.198>few <00:18:58.315>notes, <00:18:58.431>uh, <00:18:58.548>on <00:18:58.665>going

439
00:19:00.645 --> 00:19:03.165
<00:19:00.645>to <00:19:00.820>production <00:19:00.995>with <00:19:01.170>these <00:19:01.345>types <00:19:01.520>of, <00:19:01.695>uh, <00:19:01.870>of <00:19:02.045>systems.

440
00:19:04.425 --> 00:19:06.045
<00:19:04.425>So <00:19:04.891>number <00:19:05.358>one, <00:19:05.825>planning

441
00:19:06.045 --> 00:19:09.085
<00:19:06.045>and <00:19:06.325>multi-agent <00:19:06.605>collaboration <00:19:06.885>are <00:19:07.165>still <00:19:07.445>unreliable.

442
00:19:09.585 --> 00:19:12.485
<00:19:09.585>Uh, <00:19:09.701>so <00:19:09.818>most <00:19:09.935>of <00:19:10.051>the <00:19:10.168>methods <00:19:10.285>that <00:19:10.401>I've <00:19:10.518>shown <00:19:10.635>at <00:19:10.751>the <00:19:10.868>very <00:19:10.985>end,

443
00:19:12.905 --> 00:19:14.245
<00:19:12.905>you <00:19:13.146>can't <00:19:13.387>necessarily <00:19:13.628>rely <00:19:13.869>on <00:19:14.110>them.

444
00:19:14.245 --> 00:19:16.885
<00:19:14.245>They're <00:19:14.445>very <00:19:14.645>finicky. <00:19:14.844>So <00:19:15.044>you <00:19:15.245>want <00:19:15.445>to, <00:19:15.645>uh,

445
00:19:16.885 --> 00:19:19.405
<00:19:16.885>keep <00:19:17.060>your <00:19:17.235>approaches <00:19:17.410>minimum <00:19:17.585>to, <00:19:17.760>uh, <00:19:17.935>to <00:19:18.110>the <00:19:18.285>router

446
00:19:19.405 --> 00:19:21.165
<00:19:19.405>or <00:19:19.605>a <00:19:19.805>query <00:19:20.004>planner <00:19:20.205>for <00:19:20.405>the <00:19:20.605>time <00:19:20.805>being

447
00:19:21.615 --> 00:19:23.965
<00:19:21.615>until <00:19:21.742>we <00:19:21.869>have <00:19:21.996>some <00:19:22.124>more <00:19:22.251>modern <00:19:22.378>l <00:19:22.505>LMS <00:19:22.633>that <00:19:22.760>are <00:19:22.887>coming <00:19:23.015>out

448
00:19:23.965 --> 00:19:25.085
<00:19:23.965>that <00:19:24.216>can <00:19:24.469>do <00:19:24.721>better <00:19:24.973>reasoning.

449
00:19:27.125 --> 00:19:29.805
<00:19:27.125>Additionally, <00:19:27.325>you <00:19:27.525>want <00:19:27.725>to <00:19:27.925>consider <00:19:28.125>token <00:19:28.325>costs <00:19:28.525>accuracy,

450
00:19:30.185 --> 00:19:31.205
<00:19:30.185>uh, <00:19:30.644>and <00:19:31.102>latency.

451
00:19:31.205 --> 00:19:33.685
<00:19:31.205>So <00:19:31.321>these <00:19:31.438>are <00:19:31.554>the <00:19:31.671>key <00:19:31.788>metrics <00:19:31.905>that <00:19:32.021>you <00:19:32.138>need <00:19:32.254>to <00:19:32.371>keep <00:19:32.488>track <00:19:32.605>of,

452
00:19:34.275 --> 00:19:35.645
<00:19:34.275>because <00:19:34.583>if <00:19:34.891>you <00:19:35.199>don't, <00:19:35.508>well,

453
00:19:35.645 --> 00:19:37.365
<00:19:35.645>you're <00:19:35.878>gonna <00:19:36.111>have <00:19:36.345>some <00:19:36.578>very <00:19:36.811>unhappy <00:19:37.045>customers

454
00:19:38.915 --> 00:19:40.325
<00:19:38.915>when <00:19:39.168>it <00:19:39.422>comes <00:19:39.676>to <00:19:39.930>token <00:19:40.184>costs.

455
00:19:40.585 --> 00:19:42.845
<00:19:40.585>You <00:19:40.785>can <00:19:40.985>implement <00:19:41.185>every <00:19:41.385>single <00:19:41.585>pattern <00:19:41.785>out <00:19:41.985>there

456
00:19:43.145 --> 00:19:46.205
<00:19:43.145>and <00:19:43.345>build <00:19:43.545>like <00:19:43.745>this <00:19:43.945>super <00:19:44.145>solid <00:19:44.345>rag <00:19:44.545>pipeline

457
00:19:46.205 --> 00:19:47.285
<00:19:46.205>that's <00:19:46.691>like <00:19:47.177>invincible.

458
00:19:47.665 --> 00:19:50.405
<00:19:47.665>But <00:19:47.792>if <00:19:47.919>it <00:19:48.046>costs <00:19:48.174>you <00:19:48.301>like <00:19:48.428>$5 <00:19:48.555>to <00:19:48.683>run <00:19:48.810>it <00:19:48.937>each <00:19:49.065>query,

459
00:19:50.405 --> 00:19:51.965
<00:19:50.405>then <00:19:50.638>you're <00:19:50.871>not <00:19:51.105>getting <00:19:51.338>anywhere <00:19:51.571>with <00:19:51.805>this.

460
00:19:52.265 --> 00:19:53.645
<00:19:52.265>And <00:19:52.472>the <00:19:52.679>same <00:19:52.886>thing <00:19:53.093>goes <00:19:53.300>for <00:19:53.507>latency.

461
00:19:53.865 --> 00:19:56.645
<00:19:53.865>If <00:19:54.020>you're <00:19:54.176>taking <00:19:54.331>an <00:19:54.487>hour <00:19:54.642>to <00:19:54.798>run <00:19:54.953>your <00:19:55.109>pipeline, <00:19:55.265>then

462
00:19:57.265 --> 00:19:58.485
<00:19:57.265>are <00:19:57.539>you <00:19:57.814>actually <00:19:58.088>solving <00:19:58.362>something

463
00:19:58.485 --> 00:19:59.565
<00:19:58.485>if <00:19:58.679>it <00:19:58.873>gets <00:19:59.068>the <00:19:59.262>wrong <00:19:59.457>answer.

464
00:20:01.795 --> 00:20:03.395
<00:20:01.795>Additionally, <00:20:02.075>we <00:20:02.355>want <00:20:02.635>to <00:20:02.915>consider <00:20:03.195>guardrails,

465
00:20:03.395 --> 00:20:05.395
<00:20:03.395>like <00:20:03.745>rate <00:20:04.095>limiting <00:20:04.445>delegation <00:20:04.795>limits,

466
00:20:05.695 --> 00:20:08.835
<00:20:05.695>prompt <00:20:05.870>injection <00:20:06.044>defenses, <00:20:06.220>that <00:20:06.395>those <00:20:06.570>are <00:20:06.745>all <00:20:06.919>crucial, <00:20:07.095>uh,

467
00:20:08.835 --> 00:20:10.995
<00:20:08.835>and <00:20:09.115>rate <00:20:09.395>limiting <00:20:09.675>and <00:20:09.955>delegation <00:20:10.235>limits.

468
00:20:11.065 --> 00:20:13.115
<00:20:11.065>They <00:20:11.220>sort <00:20:11.376>of <00:20:11.531>go <00:20:11.687>hand <00:20:11.842>in <00:20:11.998>hand <00:20:12.153>with <00:20:12.309>token <00:20:12.465>costs.

469
00:20:13.375 --> 00:20:14.915
<00:20:13.375>If <00:20:13.573>you <00:20:13.771>get <00:20:13.969>stuck <00:20:14.167>in <00:20:14.365>an <00:20:14.562>infinite <00:20:14.761>loop,

470
00:20:14.975 --> 00:20:17.355
<00:20:14.975>you're <00:20:15.102>gonna <00:20:15.229>get <00:20:15.356>a <00:20:15.484>big <00:20:15.611>bill <00:20:15.738>and <00:20:15.865>you <00:20:15.993>don't <00:20:16.120>want <00:20:16.247>that <00:20:16.375>either.

471
00:20:18.255 --> 00:20:19.605
<00:20:18.255>Don't <00:20:18.558>blindly <00:20:18.862>trust <00:20:19.166>your <00:20:19.470>sources.

472
00:20:20.635 --> 00:20:23.845
<00:20:20.635>This <00:20:20.810>one's <00:20:20.985>straightforward, <00:20:21.160>but <00:20:21.335>I <00:20:21.510>mean, <00:20:21.685>Google <00:20:21.860>does <00:20:22.035>it.

473
00:20:23.945 --> 00:20:27.805
<00:20:23.945>So <00:20:24.085>Google's <00:20:24.225>AI <00:20:24.365>recommended <00:20:24.504>to <00:20:24.645>someone <00:20:24.784>to <00:20:24.925>put <00:20:25.064>glue <00:20:25.205>in <00:20:25.345>their,

474
00:20:27.805 --> 00:20:28.965
<00:20:27.805>uh, <00:20:28.066>in <00:20:28.327>their <00:20:28.588>pizza <00:20:28.849>sauce.

475
00:20:29.465 --> 00:20:30.885
<00:20:29.465>Uh, <00:20:29.647>I <00:20:29.830>dunno <00:20:30.012>if <00:20:30.195>if <00:20:30.377>you <00:20:30.560>saw <00:20:30.743>that,

476
00:20:30.905 --> 00:20:32.005
<00:20:30.905>but, <00:20:31.895>uh,

477
00:20:32.005 --> 00:20:34.565
<00:20:32.005>it <00:20:32.132>was <00:20:32.259>actually <00:20:32.386>from <00:20:32.514>a <00:20:32.641>Reddit <00:20:32.768>post <00:20:32.895>from <00:20:33.023>a <00:20:33.150>troll <00:20:33.277>like <00:20:33.405>10

478
00:20:34.565 --> 00:20:37.765
<00:20:34.565>years <00:20:34.705>ago, <00:20:34.845>and <00:20:34.985>it <00:20:35.125>presented <00:20:35.265>these <00:20:35.405>results <00:20:35.545>to <00:20:35.685>the <00:20:35.825>end <00:20:35.965>user.

479
00:20:37.865 --> 00:20:39.085
<00:20:37.865>So <00:20:38.048>you <00:20:38.231>don't <00:20:38.414>want <00:20:38.597>this <00:20:38.780>to <00:20:38.963>happen.

480
00:20:39.595 --> 00:20:40.965
<00:20:39.595>Make <00:20:39.841>sure <00:20:40.088>that <00:20:40.334>your <00:20:40.581>data's <00:20:40.828>clean.

481
00:20:42.585 --> 00:20:44.885
<00:20:42.585>And <00:20:42.725>last <00:20:42.865>one <00:20:43.005>is <00:20:43.145>watch <00:20:43.285>out <00:20:43.425>for <00:20:43.565>data <00:20:43.705>growth <00:20:43.845>and <00:20:43.985>bias.

482
00:20:45.385 --> 00:20:46.965
<00:20:45.385>Uh, <00:20:45.618>this <00:20:45.851>one <00:20:46.085>again, <00:20:46.318>is <00:20:46.551>pretty <00:20:46.785>straightforward,

483
00:20:47.185 --> 00:20:50.005
<00:20:47.185>but <00:20:47.385>your <00:20:47.585>data's <00:20:47.784>like <00:20:47.985>a, <00:20:48.185>a <00:20:48.385>big <00:20:48.585>haystack

484
00:20:50.305 --> 00:20:52.205
<00:20:50.305>and <00:20:50.480>you <00:20:50.655>have <00:20:50.830>a <00:20:51.005>needle <00:20:51.180>that <00:20:51.355>you're <00:20:51.530>looking <00:20:51.705>for.

485
00:20:52.705 --> 00:20:55.645
<00:20:52.705>And <00:20:52.905>as <00:20:53.105>your <00:20:53.304>pilot <00:20:53.504>of <00:20:53.705>hay <00:20:53.905>grows, <00:20:54.105>well,

486
00:20:55.785 --> 00:20:57.685
<00:20:55.785>you're <00:20:55.985>gonna <00:20:56.185>lose <00:20:56.385>sight <00:20:56.585>of <00:20:56.785>your <00:20:56.985>needle, <00:20:57.185>right?

487
00:20:58.065 --> 00:21:00.085
<00:20:58.065>So <00:20:58.240>you <00:20:58.415>gotta <00:20:58.590>make <00:20:58.765>sure <00:20:58.940>that <00:20:59.115>you're <00:20:59.290>continuously <00:20:59.465>testing

488
00:21:00.115 --> 00:21:01.765
<00:21:00.115>with <00:21:00.315>the <00:21:00.515>same <00:21:00.715>data <00:21:00.915>set <00:21:01.115>to <00:21:01.315>make <00:21:01.515>sure

489
00:21:01.765 --> 00:21:02.805
<00:21:01.765>that <00:21:01.999>the <00:21:02.233>results <00:21:02.467>don't <00:21:02.701>change.

490
00:21:05.175 --> 00:21:07.235
<00:21:05.175>And <00:21:05.315>that's <00:21:05.455>all <00:21:05.595>I <00:21:05.735>have <00:21:05.875>for <00:21:06.014>you. <00:21:06.155>Thank <00:21:06.294>you <00:21:06.435>very <00:21:06.575>much.