WEBVTT

1
00:00:05.235 --> 00:00:06.455
<00:00:05.235>Uh, <00:00:05.509>my <00:00:05.784>name <00:00:06.058>is <00:00:06.333>Phillip.

2
00:00:06.815 --> 00:00:09.735
<00:00:06.815>I <00:00:06.931>am <00:00:07.048>ai, <00:00:07.165>um, <00:00:07.281>part <00:00:07.398>of <00:00:07.515>the <00:00:07.631>AI <00:00:07.748>dero <00:00:07.865>team <00:00:07.981>at <00:00:08.098>Google <00:00:08.215>DeepMind,

3
00:00:09.735 --> 00:00:13.015
<00:00:09.735>and <00:00:09.934>especially <00:00:10.135>I <00:00:10.334>work <00:00:10.535>on <00:00:10.735>AI <00:00:10.934>developer <00:00:11.135>experience

4
00:00:13.435 --> 00:00:15.055
<00:00:13.435>and <00:00:13.668>I <00:00:13.901>think <00:00:14.135>it's <00:00:14.368>safe <00:00:14.601>to <00:00:14.835>say

5
00:00:15.055 --> 00:00:18.215
<00:00:15.055>that <00:00:15.182>we <00:00:15.309>are <00:00:15.436>currently <00:00:15.564>at <00:00:15.691>some <00:00:15.818>kind <00:00:15.945>of <00:00:16.073>turning <00:00:16.200>point <00:00:16.327>in <00:00:16.455>AI

6
00:00:18.215 --> 00:00:20.495
<00:00:18.215>and <00:00:18.448>machine <00:00:18.681>learning <00:00:18.915>where <00:00:19.148>we <00:00:19.381>see <00:00:19.615>AI

7
00:00:20.595 --> 00:00:23.655
<00:00:20.595>and <00:00:20.828>machine <00:00:21.061>learning <00:00:21.294>going <00:00:21.528>from <00:00:21.761>predicting, <00:00:21.994>trying

8
00:00:23.655 --> 00:00:26.175
<00:00:23.655>to <00:00:23.888>recognize <00:00:24.121>pattern <00:00:24.355>to <00:00:24.588>really <00:00:24.821>doing <00:00:25.055>stuff.

9
00:00:26.755 --> 00:00:29.135
<00:00:26.755>And <00:00:26.895>today <00:00:27.035>what <00:00:27.174>we <00:00:27.314>are <00:00:27.455>going <00:00:27.595>to <00:00:27.735>do, <00:00:27.875>uh, <00:00:28.015>we're <00:00:28.154>going

10
00:00:29.135 --> 00:00:31.135
<00:00:29.135>to <00:00:29.335>take <00:00:29.535>a <00:00:29.735>quick, <00:00:29.935>uh, <00:00:30.135>history <00:00:30.335>on <00:00:30.535>like

11
00:00:31.135 --> 00:00:32.735
<00:00:31.135>where <00:00:31.415>AI <00:00:31.695>agents <00:00:31.975>are <00:00:32.255>coming <00:00:32.535>from.

12
00:00:33.235 --> 00:00:35.135
<00:00:33.235>Uh, <00:00:33.468>then <00:00:33.701>we <00:00:33.935>are <00:00:34.168>going <00:00:34.401>to <00:00:34.635>define

13
00:00:35.245 --> 00:00:37.775
<00:00:35.245>what <00:00:35.445>our <00:00:35.644>agents <00:00:35.845>look <00:00:36.044>at <00:00:36.245>some <00:00:36.445>agent <00:00:36.644>patterns

14
00:00:37.775 --> 00:00:39.535
<00:00:37.775>to <00:00:38.055>help <00:00:38.335>you <00:00:38.614>build <00:00:38.894>AI <00:00:39.175>agents.

15
00:00:39.565 --> 00:00:42.535
<00:00:39.565>Then <00:00:39.765>how <00:00:39.964>can <00:00:40.165>you <00:00:40.364>evaluate <00:00:40.565>those <00:00:40.765>different <00:00:40.964>agents

16
00:00:42.875 --> 00:00:44.935
<00:00:42.875>and <00:00:43.075>then <00:00:43.275>look <00:00:43.475>at <00:00:43.675>some <00:00:43.875>practical <00:00:44.075>examples <00:00:44.275>on

17
00:00:44.935 --> 00:00:47.055
<00:00:44.935>how <00:00:45.110>you <00:00:45.285>can <00:00:45.460>use <00:00:45.635>Google <00:00:45.810>Gini <00:00:45.985>to <00:00:46.160>build <00:00:46.335>agents.

18
00:00:47.055 --> 00:00:50.055
<00:00:47.055>And <00:00:47.210>then <00:00:47.366>also <00:00:47.521>some <00:00:47.677>small <00:00:47.832>predictions <00:00:47.988>from <00:00:48.143>my <00:00:48.299>side <00:00:48.455>on

19
00:00:50.115 --> 00:00:52.335
<00:00:50.115>how <00:00:50.290>the <00:00:50.465>future, <00:00:50.640>uh, <00:00:50.815>might <00:00:50.990>going <00:00:51.165>to <00:00:51.340>look <00:00:51.515>like.

20
00:00:52.995 --> 00:00:55.055
<00:00:52.995>So <00:00:53.228>where <00:00:53.461>are <00:00:53.695>we <00:00:53.928>coming <00:00:54.161>from, <00:00:54.394>right?

21
00:00:55.205 --> 00:00:56.735
<00:00:55.205>When <00:00:55.434>you <00:00:55.664>look <00:00:55.893>at <00:00:56.123>large <00:00:56.352>language <00:00:56.582>models

22
00:00:56.755 --> 00:00:59.695
<00:00:56.755>or <00:00:56.910>language <00:00:57.066>models <00:00:57.221>in <00:00:57.377>general, <00:00:57.532>we <00:00:57.688>started <00:00:57.843>with <00:00:57.999>the <00:00:58.155>GPT

23
00:00:59.715 --> 00:01:02.735
<00:00:59.715>and <00:00:59.870>the <00:01:00.026>bird <00:01:00.181>models <00:01:00.337>where <00:01:00.492>we <00:01:00.648>had <00:01:00.803>text <00:01:00.959>completion <00:01:01.115>trying

24
00:01:02.755 --> 00:01:04.055
<00:01:02.755>to <00:01:03.145>just <00:01:03.535>complete <00:01:03.925>text.

25
00:01:04.195 --> 00:01:07.655
<00:01:04.194>We <00:01:04.322>saw <00:01:04.449>or <00:01:04.576>we <00:01:04.704>were <00:01:04.831>impressed <00:01:04.958>by <00:01:05.085>models <00:01:05.213>trying <00:01:05.340>to <00:01:05.467>write <00:01:05.595>poems

26
00:01:07.755 --> 00:01:10.095
<00:01:07.755>and <00:01:07.988>we <00:01:08.221>seesaw <00:01:08.455>or <00:01:08.688>saw <00:01:08.921>some <00:01:09.155>coherent.

27
00:01:10.355 --> 00:01:12.215
<00:01:10.355>And <00:01:10.555>then <00:01:10.755>moving <00:01:10.955>a <00:01:11.155>bit <00:01:11.355>further, <00:01:11.555>we <00:01:11.755>tried

28
00:01:12.215 --> 00:01:14.135
<00:01:12.215>to <00:01:12.448>train <00:01:12.681>those <00:01:12.915>models <00:01:13.148>to <00:01:13.381>follow <00:01:13.615>instructions.

29
00:01:14.155 --> 00:01:16.735
<00:01:14.155>So <00:01:14.330>if <00:01:14.505>we <00:01:14.680>ask <00:01:14.855>those <00:01:15.030>STEM <00:01:15.205>questions <00:01:15.380>like <00:01:15.555>what's

30
00:01:16.735 --> 00:01:17.895
<00:01:16.735>the <00:01:17.083>capital <00:01:17.431>of <00:01:17.779>Germany?

31
00:01:18.245 --> 00:01:20.495
<00:01:18.245>That <00:01:18.445>it <00:01:18.645>doesn't <00:01:18.845>continue <00:01:19.045>with <00:01:19.245>another <00:01:19.445>question <00:01:19.645>like,

32
00:01:20.495 --> 00:01:21.535
<00:01:20.495>what's <00:01:20.729>the <00:01:20.963>capital <00:01:21.197>of <00:01:21.431>France?

33
00:01:21.675 --> 00:01:23.935
<00:01:21.675>No, <00:01:21.830>we, <00:01:21.986>we <00:01:22.141>tried <00:01:22.297>or <00:01:22.452>we <00:01:22.608>trained <00:01:22.763>them <00:01:22.919>to <00:01:23.075>answer

34
00:01:24.075 --> 00:01:26.055
<00:01:24.075>or <00:01:24.250>follow <00:01:24.425>the <00:01:24.600>instructions <00:01:24.775>we <00:01:24.950>tell <00:01:25.125>them <00:01:25.300>to <00:01:25.475>do.

35
00:01:26.315 --> 00:01:27.615
<00:01:26.315>And <00:01:26.607>then <00:01:26.899>with <00:01:27.192>Jet <00:01:27.485>GPT

36
00:01:27.615 --> 00:01:30.655
<00:01:27.615>and <00:01:27.742>all <00:01:27.869>of <00:01:27.996>the <00:01:28.124>chatbot <00:01:28.251>hype, <00:01:28.378>we <00:01:28.505>moved <00:01:28.633>beyond <00:01:28.760>for <00:01:28.887>simple, <00:01:29.015>uh,

37
00:01:30.655 --> 00:01:32.935
<00:01:30.655>instruction <00:01:30.888>following <00:01:31.121>into <00:01:31.355>a <00:01:31.588>more <00:01:31.821>conversational <00:01:32.055>space

38
00:01:33.185 --> 00:01:36.295
<00:01:33.185>where <00:01:33.385>we <00:01:33.585>really <00:01:33.785>moved <00:01:33.985>beyond <00:01:34.185>like <00:01:34.385>single <00:01:34.585>instructions

39
00:01:36.295 --> 00:01:37.855
<00:01:36.295>to <00:01:36.761>like <00:01:37.228>multiterm <00:01:37.695>conversations

40
00:01:37.855 --> 00:01:39.895
<00:01:37.854>where <00:01:38.088>we <00:01:38.321>really <00:01:38.554>had <00:01:38.788>those <00:01:39.021>nice <00:01:39.255>interactions.

41
00:01:40.355 --> 00:01:43.615
<00:01:40.354>And <00:01:40.494>currently, <00:01:40.634>or <00:01:40.774>then <00:01:40.914>we <00:01:41.054>saw <00:01:41.195>the, <00:01:41.335>the <00:01:41.475>evolvement <00:01:41.615>of, <00:01:41.755>um,

42
00:01:43.615 --> 00:01:45.095
<00:01:43.615>instructions <00:01:44.281>and <00:01:44.947>functions.

43
00:01:45.515 --> 00:01:49.775
<00:01:45.515>So <00:01:45.690>we <00:01:45.865>gave <00:01:46.040>models <00:01:46.215>access <00:01:46.390>to <00:01:46.565>some <00:01:46.740>structured <00:01:46.915>information

44
00:01:49.835 --> 00:01:51.975
<00:01:49.835>and <00:01:49.990>then <00:01:50.146>instead <00:01:50.301>of <00:01:50.457>responding <00:01:50.612>with <00:01:50.768>like, <00:01:50.923>what's <00:01:51.079>the <00:01:51.235>capital

45
00:01:51.995 --> 00:01:55.455
<00:01:51.995>of <00:01:52.150>Germany <00:01:52.306>with <00:01:52.461>Berlin, <00:01:52.617>maybe <00:01:52.772>tried <00:01:52.928>to <00:01:53.083>use <00:01:53.239>some <00:01:53.395>kind

46
00:01:55.455 --> 00:01:58.015
<00:01:55.455>of <00:01:55.610>function <00:01:55.766>with, <00:01:55.921>okay, <00:01:56.077>I <00:01:56.232>need <00:01:56.388>to <00:01:56.543>use <00:01:56.699>Google <00:01:56.855>search

47
00:01:58.045 --> 00:01:59.175
<00:01:58.045>with <00:01:58.384>the <00:01:58.723>search <00:01:59.062>string.

48
00:01:59.205 --> 00:02:01.055
<00:01:59.205>Okay, <00:01:59.485>what's <00:01:59.765>the <00:02:00.045>capital? <00:02:00.325>And <00:02:00.605>then

49
00:02:01.055 --> 00:02:02.695
<00:02:01.055>where <00:02:01.288>we <00:02:01.521>are <00:02:01.755>now <00:02:01.988>like <00:02:02.221>moving <00:02:02.455>towards,

50
00:02:02.845 --> 00:02:05.895
<00:02:02.845>it's <00:02:02.972>like <00:02:03.099>really <00:02:03.226>we <00:02:03.354>want <00:02:03.481>to <00:02:03.608>combine <00:02:03.735>the <00:02:03.863>goals <00:02:03.990>then <00:02:04.117>the <00:02:04.245>new

51
00:02:05.895 --> 00:02:07.095
<00:02:05.895>capabilities <00:02:06.435>of <00:02:06.975>thinking

52
00:02:07.095 --> 00:02:09.855
<00:02:07.095>and <00:02:07.295>reasoning <00:02:07.495>models <00:02:07.695>with <00:02:07.895>those <00:02:08.095>functions <00:02:08.294>that <00:02:08.495>we

53
00:02:09.915 --> 00:02:12.935
<00:02:09.914>as <00:02:10.090>developers <00:02:10.264>and <00:02:10.440>users <00:02:10.614>only <00:02:10.789>provide <00:02:10.965>them <00:02:11.140>one <00:02:11.315>goal

54
00:02:13.055 --> 00:02:14.175
<00:02:13.055>and <00:02:13.256>then <00:02:13.458>the <00:02:13.659>model <00:02:13.861>goes <00:02:14.063>on

55
00:02:14.195 --> 00:02:15.455
<00:02:14.195>and <00:02:14.421>like <00:02:14.648>tries <00:02:14.875>to <00:02:15.102>go <00:02:15.329>in,

56
00:02:15.455 --> 00:02:18.135
<00:02:15.455>in <00:02:15.655>like <00:02:15.855>very <00:02:16.055>long <00:02:16.255>loop <00:02:16.455>using <00:02:16.655>different <00:02:16.855>tools

57
00:02:18.225 --> 00:02:19.575
<00:02:18.225>until <00:02:18.528>it <00:02:18.832>completes <00:02:19.136>that <00:02:19.440>goal.

58
00:02:19.675 --> 00:02:21.775
<00:02:19.675>So <00:02:19.850>we <00:02:20.025>are <00:02:20.200>moving <00:02:20.375>a <00:02:20.550>lot <00:02:20.725>of <00:02:20.900>like <00:02:21.075>complexity

59
00:02:21.795 --> 00:02:24.615
<00:02:21.795>and <00:02:21.950>logic <00:02:22.106>from <00:02:22.261>like <00:02:22.417>the <00:02:22.572>client <00:02:22.728>side <00:02:22.883>into <00:02:23.039>the <00:02:23.195>backend.

60
00:02:25.355 --> 00:02:26.495
<00:02:25.355>But <00:02:25.560>where <00:02:25.765>did <00:02:25.970>it <00:02:26.175>all <00:02:26.381>start?

61
00:02:26.595 --> 00:02:27.895
<00:02:26.595>So <00:02:26.985>the <00:02:27.375>first <00:02:27.765>agent

62
00:02:27.895 --> 00:02:32.215
<00:02:27.895>or <00:02:28.011>the <00:02:28.128>more <00:02:28.245>agent <00:02:28.361>paper <00:02:28.478>I <00:02:28.595>found <00:02:28.711>was <00:02:28.828>from <00:02:28.945>open <00:02:29.061>AI <00:02:29.178>in <00:02:29.295>2021

63
00:02:32.485 --> 00:02:36.055
<00:02:32.485>with <00:02:32.660>web <00:02:32.835>GBT <00:02:33.010>where <00:02:33.185>they <00:02:33.360>trained, <00:02:33.535>uh, <00:02:33.710>a <00:02:33.885>pre-check

64
00:02:36.115 --> 00:02:39.375
<00:02:36.115>or <00:02:36.348>GBT <00:02:36.581>3.5 <00:02:36.815>model <00:02:37.048>on <00:02:37.281>instruction <00:02:37.515>following

65
00:02:39.375 --> 00:02:41.575
<00:02:39.375>where <00:02:39.550>the <00:02:39.725>model <00:02:39.900>tried <00:02:40.075>to <00:02:40.250>navigate <00:02:40.425>through <00:02:40.600>the <00:02:40.775>browser,

66
00:02:41.985 --> 00:02:44.095
<00:02:41.985>where <00:02:42.185>they <00:02:42.385>gave <00:02:42.585>them <00:02:42.785>a <00:02:42.985>simple <00:02:43.185>instruction, <00:02:43.385>okay,

67
00:02:44.095 --> 00:02:45.695
<00:02:44.095>like <00:02:44.295>can <00:02:44.495>you <00:02:44.695>find <00:02:44.895>the <00:02:45.095>capital <00:02:45.295>of <00:02:45.495>Germany?

68
00:02:45.755 --> 00:02:48.575
<00:02:45.755>And <00:02:45.910>then, <00:02:46.066>uh, <00:02:46.221>GBT <00:02:46.377>tries <00:02:46.532>to <00:02:46.688>navigate <00:02:46.843>through <00:02:46.999>the <00:02:47.155>browser,

69
00:02:48.625 --> 00:02:50.935
<00:02:48.625>going <00:02:48.800>to <00:02:48.975>Google <00:02:49.150>Google <00:02:49.325>search, <00:02:49.500>opening <00:02:49.675>different <00:02:49.850>pages <00:02:50.025>like

70
00:02:50.965 --> 00:02:53.375
<00:02:50.965>Wikipedia <00:02:51.245>and <00:02:51.525>trying <00:02:51.805>to <00:02:52.085>extract <00:02:52.365>information.

71
00:02:53.715 --> 00:02:55.535
<00:02:53.715>And <00:02:53.948>another <00:02:54.181>paper <00:02:54.415>was <00:02:54.648>from, <00:02:54.881>uh, <00:02:55.115>meta

72
00:02:55.605 --> 00:02:58.935
<00:02:55.605>with <00:02:55.760>tool <00:02:55.916>former <00:02:56.071>in <00:02:56.227>2023 <00:02:56.382>where <00:02:56.538>they <00:02:56.693>trained <00:02:56.849>a <00:02:57.005>model

73
00:02:59.025 --> 00:03:00.485
<00:02:59.025>to <00:02:59.463>use <00:02:59.901>any <00:03:00.339>tool.

74
00:03:00.865 --> 00:03:04.725
<00:03:00.865>Um, <00:03:01.040>and <00:03:01.215>by <00:03:01.390>simply <00:03:01.565>continuing <00:03:01.740>the <00:03:01.915>tax <00:03:02.090>completion <00:03:02.265>model

75
00:03:04.735 --> 00:03:06.765
<00:03:04.735>where <00:03:05.015>they <00:03:05.295>tried <00:03:05.575>to <00:03:05.855>inject, <00:03:06.135>uh,

76
00:03:06.765 --> 00:03:09.165
<00:03:06.765>different <00:03:07.114>sections <00:03:07.464>into <00:03:07.815>the <00:03:08.165>generation

77
00:03:09.345 --> 00:03:12.165
<00:03:09.345>to <00:03:09.472>help <00:03:09.599>the <00:03:09.726>model <00:03:09.854>predict, <00:03:09.981>okay, <00:03:10.108>when <00:03:10.235>should <00:03:10.363>it <00:03:10.490>use <00:03:10.617>a <00:03:10.745>tool,

78
00:03:12.165 --> 00:03:13.325
<00:03:12.165>when <00:03:12.373>should <00:03:12.582>it <00:03:12.791>use <00:03:13.000>a <00:03:13.209>calculator?

79
00:03:14.265 --> 00:03:18.325
<00:03:14.265>And <00:03:14.404>then <00:03:14.545>2020, <00:03:14.684>uh, <00:03:14.825>two, <00:03:14.964>uh, <00:03:15.105>we <00:03:15.244>got <00:03:15.385>the <00:03:15.524>react <00:03:15.665>paper,

80
00:03:18.615 --> 00:03:20.445
<00:03:18.615>which <00:03:18.848>was <00:03:19.081>the <00:03:19.315>first <00:03:19.548>paper <00:03:19.781>which <00:03:20.015>tried

81
00:03:20.445 --> 00:03:24.285
<00:03:20.445>to <00:03:20.572>combine <00:03:20.699>the <00:03:20.826>reasoning <00:03:20.954>of <00:03:21.081>an <00:03:21.208>LLM <00:03:21.335>with <00:03:21.463>some <00:03:21.590>kind <00:03:21.717>of, <00:03:21.845>uh,

82
00:03:24.305 --> 00:03:25.325
<00:03:24.305>action <00:03:25.222>obsession.

83
00:03:25.585 --> 00:03:27.285
<00:03:25.585>So <00:03:25.785>we <00:03:25.985>had <00:03:26.185>our, <00:03:26.385>we <00:03:26.585>have <00:03:26.785>our <00:03:26.985>prompt.

84
00:03:27.385 --> 00:03:29.485
<00:03:27.385>In <00:03:27.560>this <00:03:27.734>case <00:03:27.910>we <00:03:28.084>have, <00:03:28.260>okay, <00:03:28.435>who <00:03:28.609>is <00:03:28.785>Olivia's?

85
00:03:29.485 --> 00:03:30.805
<00:03:29.485>Uh, <00:03:29.881>Olivia <00:03:30.277>Wise's <00:03:30.673>boyfriend,

86
00:03:30.805 --> 00:03:33.205
<00:03:30.805>and <00:03:30.960>what <00:03:31.116>is <00:03:31.271>the <00:03:31.427>current <00:03:31.582>age <00:03:31.738>race <00:03:31.893>to <00:03:32.049>the <00:03:32.205>um,

87
00:03:34.075 --> 00:03:35.405
<00:03:34.075>zero <00:03:34.474>point <00:03:34.873>23 <00:03:35.272>power?

88
00:03:35.595 --> 00:03:37.445
<00:03:35.595>This <00:03:35.875>wouldn't <00:03:36.155>be <00:03:36.435>like <00:03:36.715>very <00:03:36.995>simple

89
00:03:38.145 --> 00:03:39.885
<00:03:38.145>if <00:03:38.378>you <00:03:38.611>have <00:03:38.845>like <00:03:39.078>one <00:03:39.311>tool <00:03:39.545>call.

90
00:03:39.905 --> 00:03:41.565
<00:03:39.905>So <00:03:40.138>you <00:03:40.371>need <00:03:40.605>some <00:03:40.838>kind <00:03:41.071>of <00:03:41.305>reasoning

91
00:03:41.565 --> 00:03:42.925
<00:03:41.565>and <00:03:41.871>like <00:03:42.177>a <00:03:42.483>multi-step <00:03:42.789>approach.

92
00:03:43.175 --> 00:03:45.365
<00:03:43.175>First <00:03:43.350>you <00:03:43.525>need <00:03:43.700>to <00:03:43.875>know, <00:03:44.050>okay, <00:03:44.225>who's <00:03:44.400>the <00:03:44.575>boyfriend,

93
00:03:45.795 --> 00:03:47.045
<00:03:45.795>what <00:03:46.076>is <00:03:46.357>their <00:03:46.638>current <00:03:46.920>age?

94
00:03:47.105 --> 00:03:49.205
<00:03:47.105>And <00:03:47.260>then <00:03:47.416>you <00:03:47.571>also <00:03:47.727>need <00:03:47.882>to <00:03:48.038>do <00:03:48.193>the <00:03:48.349>mouth <00:03:48.505>calculation.

95
00:03:49.205 --> 00:03:50.685
<00:03:49.204>And <00:03:49.471>in <00:03:49.737>the <00:03:50.004>React <00:03:50.270>paper, <00:03:50.537>um,

96
00:03:50.685 --> 00:03:53.525
<00:03:50.685>they <00:03:50.918>introduced <00:03:51.151>this <00:03:51.385>thought <00:03:51.618>action <00:03:51.851>observe <00:03:52.085>pattern,

97
00:03:53.735 --> 00:03:57.805
<00:03:53.735>which <00:03:53.890>basically <00:03:54.046>tries <00:03:54.201>to <00:03:54.357>mimic <00:03:54.512>this <00:03:54.668>kind <00:03:54.823>of <00:03:54.979>reasoning <00:03:55.135>loop,

98
00:03:57.805 --> 00:03:59.845
<00:03:57.805>which <00:03:57.960>we <00:03:58.116>see <00:03:58.271>today <00:03:58.427>in <00:03:58.582>all <00:03:58.738>of <00:03:58.893>the <00:03:59.049>different <00:03:59.205>models.

99
00:04:00.385 --> 00:04:02.965
<00:04:00.385>And <00:04:00.512>then <00:04:00.639>looking <00:04:00.766>at <00:04:00.894>the <00:04:01.021>current <00:04:01.148>state <00:04:01.275>where <00:04:01.403>we <00:04:01.530>have <00:04:01.657>tool <00:04:01.785>use

100
00:04:02.965 --> 00:04:04.645
<00:04:02.965>and <00:04:03.198>function <00:04:03.431>calling, <00:04:03.665>where <00:04:03.898>we <00:04:04.131>moved <00:04:04.365>all

101
00:04:04.645 --> 00:04:06.365
<00:04:04.645>of <00:04:04.878>the <00:04:05.111>learnings <00:04:05.345>from <00:04:05.578>these <00:04:05.811>react <00:04:06.045>pattern

102
00:04:07.125 --> 00:04:09.685
<00:04:07.125>directly <00:04:07.265>into <00:04:07.405>the <00:04:07.545>model <00:04:07.685>and <00:04:07.825>are <00:04:07.965>now <00:04:08.105>using <00:04:08.245>a <00:04:08.385>more <00:04:08.525>structured

103
00:04:09.705 --> 00:04:13.885
<00:04:09.705>output <00:04:09.860>to, <00:04:10.016>um, <00:04:10.171>help <00:04:10.327>us <00:04:10.482>call <00:04:10.638>external <00:04:10.793>APIs <00:04:10.949>and <00:04:11.105>models.

104
00:04:14.225 --> 00:04:16.685
<00:04:14.225>It <00:04:14.458>is <00:04:14.691>used <00:04:14.924>to <00:04:15.158>augment <00:04:15.391>the <00:04:15.625>knowledge.

105
00:04:16.745 --> 00:04:20.005
<00:04:16.745>So <00:04:16.945>all <00:04:17.144>LMS <00:04:17.345>have <00:04:17.545>uh, <00:04:17.745>basically <00:04:17.945>knowledge <00:04:18.144>cutoff,

106
00:04:20.005 --> 00:04:22.565
<00:04:20.005>so <00:04:20.204>they <00:04:20.404>don't <00:04:20.605>know <00:04:20.805>about <00:04:21.005>events <00:04:21.204>that <00:04:21.404>happened

107
00:04:22.565 --> 00:04:23.645
<00:04:22.565>after <00:04:22.889>they <00:04:23.212>were <00:04:23.537>drained.

108
00:04:23.645 --> 00:04:25.525
<00:04:23.645>So <00:04:23.844>that's <00:04:24.044>a <00:04:24.245>way <00:04:24.445>we <00:04:24.645>can <00:04:24.844>extend <00:04:25.044>them.

109
00:04:25.585 --> 00:04:27.365
<00:04:25.585>We <00:04:25.784>can <00:04:25.984>give <00:04:26.185>them <00:04:26.385>access <00:04:26.585>to <00:04:26.784>different <00:04:26.984>tools

110
00:04:27.385 --> 00:04:29.125
<00:04:27.385>or <00:04:27.851>specialties <00:04:28.318>like <00:04:28.784>calculators

111
00:04:29.425 --> 00:04:31.485
<00:04:29.425>or <00:04:29.625>also <00:04:29.825>now <00:04:30.025>take <00:04:30.225>actions <00:04:30.425>for <00:04:30.625>us, <00:04:30.825>um,

112
00:04:31.485 --> 00:04:33.165
<00:04:31.485>calling <00:04:31.765>an <00:04:32.045>API <00:04:32.325>or <00:04:32.605>other <00:04:32.885>services.

113
00:04:34.065 --> 00:04:35.085
<00:04:34.065>So <00:04:34.294>what <00:04:34.524>is <00:04:34.753>an <00:04:34.983>agent?

114
00:04:35.525 --> 00:04:39.125
<00:04:35.525>I <00:04:35.641>like <00:04:35.758>to <00:04:35.875>define <00:04:35.991>an <00:04:36.108>agent <00:04:36.224>as <00:04:36.341>a <00:04:36.458>system <00:04:36.575>that <00:04:36.691>uses <00:04:36.808>an <00:04:36.924>LLM

115
00:04:39.185 --> 00:04:41.765
<00:04:39.185>to <00:04:39.385>decide <00:04:39.585>the <00:04:39.785>control <00:04:39.985>flow <00:04:40.185>of <00:04:40.385>an <00:04:40.585>application.

116
00:04:42.105 --> 00:04:44.445
<00:04:42.105>And <00:04:42.305>it's <00:04:42.505>very <00:04:42.705>important <00:04:42.905>that <00:04:43.105>we <00:04:43.305>make <00:04:43.505>sure

117
00:04:44.445 --> 00:04:46.485
<00:04:44.445>that <00:04:44.724>it's <00:04:45.005>deciding <00:04:45.284>a <00:04:45.565>control <00:04:45.844>flow

118
00:04:46.485 --> 00:04:49.325
<00:04:46.485>because <00:04:46.685>if <00:04:46.885>we <00:04:47.085>compare <00:04:47.285>it <00:04:47.485>to <00:04:47.685>like <00:04:47.885>workflows

119
00:04:49.325 --> 00:04:50.445
<00:04:49.325>or <00:04:49.829>other <00:04:50.332>scenarios,

120
00:04:50.945 --> 00:04:53.565
<00:04:50.945>an <00:04:51.120>agent <00:04:51.295>really <00:04:51.469>decides <00:04:51.645>what <00:04:51.820>it <00:04:51.995>wants <00:04:52.170>to <00:04:52.344>do.

121
00:04:53.865 --> 00:04:56.285
<00:04:53.865>And <00:04:54.040>if <00:04:54.215>we <00:04:54.390>included <00:04:54.565>all <00:04:54.740>of <00:04:54.915>what <00:04:55.090>we <00:04:55.265>know,

122
00:04:56.285 --> 00:04:58.285
<00:04:56.284>so <00:04:56.460>we <00:04:56.635>have <00:04:56.809>the <00:04:56.984>reasoning, <00:04:57.159>which <00:04:57.335>is <00:04:57.510>our <00:04:57.684>LLM,

123
00:04:58.415 --> 00:05:01.765
<00:04:58.415>which <00:04:58.570>is <00:04:58.726>the <00:04:58.881>brain <00:04:59.037>of <00:04:59.192>our <00:04:59.348>agent, <00:04:59.503>which <00:04:59.659>thinks <00:04:59.815>about

124
00:05:01.765 --> 00:05:03.125
<00:05:01.765>what <00:05:02.009>the <00:05:02.254>user <00:05:02.499>wants <00:05:02.744>to <00:05:02.989>achieve

125
00:05:03.145 --> 00:05:05.005
<00:05:03.145>and <00:05:03.320>he <00:05:03.495>evaluates <00:05:03.669>the <00:05:03.844>different <00:05:04.020>tools <00:05:04.195>it <00:05:04.370>could <00:05:04.544>use.

126
00:05:05.025 --> 00:05:06.925
<00:05:05.025>We <00:05:05.304>have <00:05:05.585>context <00:05:05.864>and <00:05:06.145>memory <00:05:06.424>here.

127
00:05:06.925 --> 00:05:10.045
<00:05:06.925>We <00:05:07.100>separate <00:05:07.275>between <00:05:07.450>short <00:05:07.625>term <00:05:07.800>memory <00:05:07.975>and <00:05:08.150>long-term <00:05:08.325>memory.

128
00:05:10.045 --> 00:05:13.925
<00:05:10.045>Short-term <00:05:10.200>memory <00:05:10.356>is <00:05:10.511>basically <00:05:10.667>the <00:05:10.822>context <00:05:10.978>size <00:05:11.133>of <00:05:11.289>our <00:05:11.445>LLM

129
00:05:14.145 --> 00:05:16.085
<00:05:14.145>and <00:05:14.495>long-term <00:05:14.844>memory <00:05:15.195>are <00:05:15.544>preferences

130
00:05:16.465 --> 00:05:20.285
<00:05:16.465>and <00:05:16.640>other <00:05:16.815>like <00:05:16.989>information <00:05:17.164>which <00:05:17.340>might <00:05:17.515>happen <00:05:17.690>from <00:05:17.864>previous

131
00:05:20.295 --> 00:05:23.165
<00:05:20.295>tasks, <00:05:20.495>which <00:05:20.695>we <00:05:20.895>include <00:05:21.095>externally <00:05:21.295>into <00:05:21.495>our <00:05:21.695>context.

132
00:05:23.345 --> 00:05:25.365
<00:05:23.345>So <00:05:23.545>if <00:05:23.745>you <00:05:23.945>have <00:05:24.145>a <00:05:24.345>customer <00:05:24.545>support <00:05:24.745>agent

133
00:05:25.365 --> 00:05:30.285
<00:05:25.365>and <00:05:25.505>the <00:05:25.645>same <00:05:25.785>customer <00:05:25.925>has <00:05:26.065>like <00:05:26.205>asked <00:05:26.345>questions <00:05:26.485>one <00:05:26.625>day <00:05:26.765>prior

134
00:05:30.285 --> 00:05:32.885
<00:05:30.285>to <00:05:30.485>the <00:05:30.685>current <00:05:30.885>uh, <00:05:31.085>interaction, <00:05:31.285>we <00:05:31.485>might <00:05:31.685>want

135
00:05:32.885 --> 00:05:34.605
<00:05:32.885>to <00:05:33.164>include <00:05:33.445>some <00:05:33.724>information <00:05:34.005>about <00:05:34.284>this

136
00:05:34.605 --> 00:05:36.045
<00:05:34.605>conversation <00:05:34.929>into <00:05:35.253>our <00:05:35.577>agent <00:05:35.901>flow.

137
00:05:36.045 --> 00:05:37.085
<00:05:36.045>And <00:05:36.279>that's <00:05:36.513>the <00:05:36.747>long-term <00:05:36.981>memory

138
00:05:37.385 --> 00:05:39.165
<00:05:37.385>and <00:05:37.618>of <00:05:37.851>course <00:05:38.085>tool <00:05:38.318>that's <00:05:38.551>our <00:05:38.784>hands

139
00:05:39.185 --> 00:05:42.165
<00:05:39.185>or <00:05:39.325>sense <00:05:39.465>of <00:05:39.605>the <00:05:39.745>agents <00:05:39.885>which <00:05:40.025>allows <00:05:40.165>it <00:05:40.305>to <00:05:40.445>call <00:05:40.585>APIs

140
00:05:42.165 --> 00:05:43.285
<00:05:42.165>and <00:05:42.417>do <00:05:42.669>actions <00:05:42.921>for <00:05:43.173>us.

141
00:05:44.825 --> 00:05:46.885
<00:05:44.825>And <00:05:45.175>comparing <00:05:45.525>workflows <00:05:45.875>with <00:05:46.224>agents,

142
00:05:47.005 --> 00:05:50.245
<00:05:47.005>I <00:05:47.160>said <00:05:47.316>already <00:05:47.471>an <00:05:47.627>agent <00:05:47.782>really <00:05:47.938>decides <00:05:48.093>the <00:05:48.249>control <00:05:48.405>flow

143
00:05:50.305 --> 00:05:53.685
<00:05:50.305>and <00:05:50.460>a <00:05:50.616>workflow <00:05:50.771>is <00:05:50.927>basically <00:05:51.082>a <00:05:51.238>predefined <00:05:51.393>sequence <00:05:51.549>of <00:05:51.705>steps.

144
00:05:53.985 --> 00:05:56.765
<00:05:53.985>So <00:05:54.125>at <00:05:54.265>the <00:05:54.405>top <00:05:54.545>we <00:05:54.685>have <00:05:54.825>a <00:05:54.965>very <00:05:55.105>simple <00:05:55.245>visual <00:05:55.385>on

145
00:05:56.785 --> 00:05:57.805
<00:05:56.784>how <00:05:57.091>our <00:05:57.397>workflow <00:05:57.703>is.

146
00:05:58.145 --> 00:06:01.645
<00:05:58.145>And <00:05:58.300>I <00:05:58.456>would <00:05:58.611>say <00:05:58.767>most <00:05:58.922>commonly <00:05:59.078>currently <00:05:59.233>the <00:05:59.389>AI <00:05:59.544>applications

147
00:06:01.665 --> 00:06:03.925
<00:06:01.665>we <00:06:01.945>build <00:06:02.225>are <00:06:02.505>80, <00:06:02.785>90% <00:06:03.065>workflows.

148
00:06:03.925 --> 00:06:07.645
<00:06:03.925>We <00:06:04.052>have <00:06:04.179>some <00:06:04.306>idea <00:06:04.434>about <00:06:04.561>doing <00:06:04.688>a <00:06:04.815>web <00:06:04.943>search <00:06:05.070>where <00:06:05.197>we <00:06:05.325>want

149
00:06:07.645 --> 00:06:10.885
<00:06:07.645>to <00:06:07.761>run <00:06:07.878>a <00:06:07.995>query <00:06:08.111>first, <00:06:08.228>look <00:06:08.344>at <00:06:08.461>the <00:06:08.578>output, <00:06:08.695>run <00:06:08.811>a <00:06:08.928>second <00:06:09.044>query

150
00:06:10.885 --> 00:06:14.565
<00:06:10.885>where <00:06:11.060>we <00:06:11.235>have <00:06:11.409>this <00:06:11.585>really <00:06:11.760>predefined <00:06:11.935>step <00:06:12.110>of <00:06:12.284>actions

151
00:06:14.585 --> 00:06:16.165
<00:06:14.585>and <00:06:14.818>calls <00:06:15.051>we <00:06:15.284>are <00:06:15.518>going <00:06:15.751>to <00:06:15.984>do.

152
00:06:16.505 --> 00:06:19.925
<00:06:16.505>And <00:06:16.660>with <00:06:16.816>an <00:06:16.971>agent, <00:06:17.127>we <00:06:17.282>have <00:06:17.438>this <00:06:17.593>basically <00:06:17.749>dynamic <00:06:17.905>control

153
00:06:19.925 --> 00:06:22.805
<00:06:19.925>flow <00:06:20.100>where <00:06:20.275>the <00:06:20.450>LLM <00:06:20.625>decides, <00:06:20.800>okay, <00:06:20.975>do <00:06:21.150>I <00:06:21.325>need

154
00:06:22.805 --> 00:06:24.045
<00:06:22.805>to <00:06:23.177>call <00:06:23.549>another <00:06:23.921>tool

155
00:06:24.425 --> 00:06:26.925
<00:06:24.425>or <00:06:24.625>is <00:06:24.825>the <00:06:25.025>information <00:06:25.225>I <00:06:25.425>have <00:06:25.625>available <00:06:25.825>enough

156
00:06:26.945 --> 00:06:28.285
<00:06:26.945>to <00:06:27.246>answer <00:06:27.548>the <00:06:27.849>user <00:06:28.151>questions?

157
00:06:28.665 --> 00:06:33.565
<00:06:28.665>So <00:06:28.820>very <00:06:28.976>important, <00:06:29.131>that's <00:06:29.287>a <00:06:29.442>workflow <00:06:29.598>that's <00:06:29.753>an <00:06:29.909>agent. <00:06:30.065>Okay?

158
00:06:33.645 --> 00:06:36.285
<00:06:33.645>Again, <00:06:33.924>workflow, <00:06:34.205>we <00:06:34.484>don't <00:06:34.765>have <00:06:35.044>any

159
00:06:36.925 --> 00:06:39.245
<00:06:36.925>deciding <00:06:37.275>control <00:06:37.625>flow <00:06:37.975>from <00:06:38.325>L-M-L-L-L-M,

160
00:06:39.245 --> 00:06:43.525
<00:06:39.245>we <00:06:39.420>have <00:06:39.595>predefined <00:06:39.770>que <00:06:39.945>in <00:06:40.120>aggregator <00:06:40.295>and <00:06:40.470>output <00:06:40.645>agent.

161
00:06:43.745 --> 00:06:47.165
<00:06:43.745>We <00:06:43.872>have <00:06:43.999>an <00:06:44.126>A <00:06:44.254>query, <00:06:44.381>an <00:06:44.508>LLM <00:06:44.635>makes <00:06:44.763>a <00:06:44.890>decision <00:06:45.017>on <00:06:45.145>what

162
00:06:47.165 --> 00:06:49.045
<00:06:47.165>to <00:06:47.365>do <00:06:47.565>next <00:06:47.765>and <00:06:47.965>then <00:06:48.165>a <00:06:48.365>final <00:06:48.565>output.

163
00:06:50.065 --> 00:06:52.085
<00:06:50.065>So <00:06:50.415>what <00:06:50.765>are <00:06:51.115>genic <00:06:51.465>patterns?

164
00:06:52.085 --> 00:06:56.325
<00:06:52.085>Um, <00:06:52.260>anic <00:06:52.435>patterns <00:06:52.609>help <00:06:52.784>us <00:06:52.960>to <00:06:53.135>more <00:06:53.310>easily <00:06:53.484>create

165
00:06:56.545 --> 00:06:59.645
<00:06:56.545>or <00:06:56.825>define <00:06:57.105>different <00:06:57.385>agent, <00:06:57.665>uh, <00:06:57.945>architectures.

166
00:06:59.905 --> 00:07:02.285
<00:06:59.905>And <00:07:00.104>like <00:07:00.304>very <00:07:00.505>important, <00:07:00.705>those <00:07:00.905>patterns <00:07:01.104>are <00:07:01.304>not

167
00:07:02.955 --> 00:07:05.205
<00:07:02.955>like <00:07:03.130>you <00:07:03.305>cannot <00:07:03.479>only <00:07:03.655>use <00:07:03.830>one <00:07:04.005>at <00:07:04.180>a <00:07:04.354>time.

168
00:07:05.235 --> 00:07:07.125
<00:07:05.235>It's <00:07:05.468>really <00:07:05.701>that <00:07:05.935>you <00:07:06.168>can <00:07:06.401>compose <00:07:06.635>them.

169
00:07:07.125 --> 00:07:08.605
<00:07:07.125>And <00:07:07.315>if <00:07:07.505>you <00:07:07.695>are <00:07:07.886>starting <00:07:08.076>to <00:07:08.266>build <00:07:08.457>agents

170
00:07:08.865 --> 00:07:11.125
<00:07:08.865>or <00:07:09.098>already <00:07:09.331>building <00:07:09.565>agents, <00:07:09.798>you <00:07:10.031>will <00:07:10.265>see

171
00:07:11.235 --> 00:07:14.045
<00:07:11.235>that <00:07:11.390>you <00:07:11.546>might <00:07:11.701>already <00:07:11.857>use <00:07:12.012>those <00:07:12.168>patterns <00:07:12.323>kind <00:07:12.479>of <00:07:12.635>natively

172
00:07:14.105 --> 00:07:15.805
<00:07:14.105>and <00:07:14.338>you <00:07:14.571>can <00:07:14.805>like <00:07:15.038>compose <00:07:15.271>them <00:07:15.505>together.

173
00:07:16.285 --> 00:07:18.325
<00:07:16.285>A <00:07:16.485>very <00:07:16.685>common <00:07:16.885>pattern <00:07:17.085>is <00:07:17.285>the <00:07:17.485>reflection <00:07:17.685>pattern.

174
00:07:18.905 --> 00:07:21.565
<00:07:18.905>Um, <00:07:19.044>it's <00:07:19.184>maybe <00:07:19.325>something <00:07:19.465>you <00:07:19.604>might <00:07:19.744>have <00:07:19.885>already <00:07:20.025>used <00:07:20.164>in <00:07:20.304>Jet

175
00:07:21.685 --> 00:07:23.085
<00:07:21.685>GBT <00:07:21.937>where <00:07:22.189>you <00:07:22.441>ask <00:07:22.693>it <00:07:22.945>to

176
00:07:23.085 --> 00:07:25.685
<00:07:23.085>or <00:07:23.284>ask <00:07:23.484>an <00:07:23.685>LLM <00:07:23.885>to <00:07:24.085>generate <00:07:24.284>some, <00:07:24.484>uh,

177
00:07:25.685 --> 00:07:27.165
<00:07:25.685>marketing <00:07:26.129>text <00:07:26.573>some <00:07:27.017>story.

178
00:07:27.585 --> 00:07:30.125
<00:07:27.585>And <00:07:27.784>then <00:07:27.984>instead <00:07:28.185>of <00:07:28.385>using <00:07:28.585>the <00:07:28.784>output <00:07:28.984>directly,

179
00:07:30.225 --> 00:07:33.005
<00:07:30.225>you <00:07:30.425>basically <00:07:30.625>ask <00:07:30.825>the <00:07:31.025>LLM <00:07:31.225>again <00:07:31.425>to <00:07:31.625>reflect

180
00:07:33.025 --> 00:07:35.405
<00:07:33.025>and <00:07:33.200>to <00:07:33.375>critique <00:07:33.549>the <00:07:33.724>output <00:07:33.900>of <00:07:34.075>the <00:07:34.250>first <00:07:34.424>LLM

181
00:07:35.625 --> 00:07:36.965
<00:07:35.625>and <00:07:35.926>ask <00:07:36.228>it <00:07:36.529>for <00:07:36.831>suggestions.

182
00:07:36.965 --> 00:07:39.885
<00:07:36.965>Okay, <00:07:37.120>what <00:07:37.276>can <00:07:37.431>you <00:07:37.587>improve, <00:07:37.742>uh, <00:07:37.898>on <00:07:38.053>the <00:07:38.209>first <00:07:38.365>response?

183
00:07:39.885 --> 00:07:43.405
<00:07:39.885>And <00:07:40.012>then <00:07:40.139>you <00:07:40.266>have <00:07:40.394>this <00:07:40.521>kind <00:07:40.648>of, <00:07:40.775>um, <00:07:40.903>loop <00:07:41.030>where <00:07:41.157>one <00:07:41.284>LLM

184
00:07:43.405 --> 00:07:46.965
<00:07:43.405>or <00:07:43.560>the <00:07:43.716>same <00:07:43.871>LLM <00:07:44.027>basically <00:07:44.182>tries <00:07:44.338>to <00:07:44.493>improve <00:07:44.649>by <00:07:44.804>itself,

185
00:07:47.265 --> 00:07:48.645
<00:07:47.265>um, <00:07:47.679>the <00:07:48.092>the <00:07:48.507>output.

186
00:07:48.865 --> 00:07:51.685
<00:07:48.865>And <00:07:48.992>at <00:07:49.119>the <00:07:49.246>end <00:07:49.374>you <00:07:49.501>might <00:07:49.628>have <00:07:49.755>a <00:07:49.883>final <00:07:50.010>output <00:07:50.137>depending <00:07:50.265>on

187
00:07:51.685 --> 00:07:54.645
<00:07:51.685>what <00:07:51.885>the <00:07:52.085>LRD <00:07:52.285>reflection <00:07:52.485>LLM <00:07:52.685>decides <00:07:52.885>if <00:07:53.085>the

188
00:07:54.645 --> 00:07:55.765
<00:07:54.645>output <00:07:54.897>is <00:07:55.149>correct <00:07:55.401>or <00:07:55.652>not.

189
00:07:55.765 --> 00:07:57.725
<00:07:55.765>Correct. <00:07:56.044>Very <00:07:56.325>helpful <00:07:56.604>for <00:07:56.885>writing <00:07:57.164>code.

190
00:07:57.985 --> 00:07:59.645
<00:07:57.985>Um, <00:07:58.335>content <00:07:58.685>creation <00:07:59.035>of <00:07:59.385>course.

191
00:08:00.625 --> 00:08:01.965
<00:08:00.625>And <00:08:00.797>then <00:08:00.969>we <00:08:01.141>have <00:08:01.314>the <00:08:01.486>tool <00:08:01.658>use <00:08:01.830>pattern.

192
00:08:02.445 --> 00:08:04.365
<00:08:02.445>I <00:08:02.645>would <00:08:02.844>assume <00:08:03.045>that's <00:08:03.245>probably <00:08:03.445>the <00:08:03.645>most, <00:08:03.844>uh,

193
00:08:04.725 --> 00:08:06.645
<00:08:04.725>commonly <00:08:05.005>known <00:08:05.285>pattern <00:08:05.565>for <00:08:05.845>building <00:08:06.125>agent.

194
00:08:06.935 --> 00:08:10.845
<00:08:06.935>Where <00:08:07.062>we <00:08:07.189>have <00:08:07.316>our <00:08:07.444>LLM, <00:08:07.571>we <00:08:07.698>provide, <00:08:07.825>um, <00:08:07.953>some <00:08:08.080>kind <00:08:08.207>of, <00:08:08.335>uh,

195
00:08:11.205 --> 00:08:13.205
<00:08:11.205>function <00:08:11.484>or <00:08:11.765>tool <00:08:12.044>use <00:08:12.325>declarations <00:08:12.604>on

196
00:08:13.205 --> 00:08:16.005
<00:08:13.205>how <00:08:13.438>the <00:08:13.671>LLM <00:08:13.905>can <00:08:14.138>access <00:08:14.371>external <00:08:14.604>APIs.

197
00:08:16.005 --> 00:08:18.765
<00:08:16.005>That's <00:08:16.238>mostly <00:08:16.471>done <00:08:16.705>by <00:08:16.938>defining <00:08:17.171>adjacent <00:08:17.405>schema.

198
00:08:18.785 --> 00:08:22.325
<00:08:18.785>So <00:08:18.912>if <00:08:19.039>you <00:08:19.166>would <00:08:19.294>like <00:08:19.421>to <00:08:19.548>integrate <00:08:19.675>Google <00:08:19.803>search <00:08:19.930>with <00:08:20.057>an <00:08:20.185>LLM,

199
00:08:22.545 --> 00:08:24.965
<00:08:22.545>you <00:08:22.778>would <00:08:23.011>have <00:08:23.245>to <00:08:23.478>define <00:08:23.711>adjacent <00:08:23.945>schema

200
00:08:24.965 --> 00:08:26.365
<00:08:24.965>with <00:08:25.385>a <00:08:25.805>title <00:08:26.225>description

201
00:08:26.365 --> 00:08:28.765
<00:08:26.365>and <00:08:26.565>the <00:08:26.765>input <00:08:26.965>parameter, <00:08:27.165>which <00:08:27.365>also <00:08:27.565>have <00:08:27.765>description.

202
00:08:29.025 --> 00:08:32.165
<00:08:29.025>You <00:08:29.200>provide <00:08:29.375>this <00:08:29.549>description <00:08:29.724>when <00:08:29.900>you <00:08:30.075>send <00:08:30.250>a <00:08:30.424>request

203
00:08:32.165 --> 00:08:34.205
<00:08:32.164>to <00:08:32.398>the <00:08:32.631>LLM <00:08:32.865>alongside <00:08:33.098>the <00:08:33.331>user <00:08:33.564>query.

204
00:08:34.545 --> 00:08:38.285
<00:08:34.544>And <00:08:34.672>then <00:08:34.799>the <00:08:34.926>LLM <00:08:35.054>decides <00:08:35.181>if <00:08:35.308>I <00:08:35.435>need <00:08:35.563>to <00:08:35.690>use <00:08:35.817>the <00:08:35.944>tool

205
00:08:38.285 --> 00:08:40.925
<00:08:38.284>to <00:08:38.459>Google <00:08:38.635>search <00:08:38.809>or <00:08:38.985>if <00:08:39.159>I <00:08:39.334>can <00:08:39.510>just <00:08:39.684>respond.

206
00:08:40.985 --> 00:08:44.205
<00:08:40.985>So <00:08:41.125>if <00:08:41.265>a <00:08:41.405>user <00:08:41.544>on <00:08:41.685>LLM <00:08:41.825>has <00:08:41.965>access <00:08:42.105>to <00:08:42.245>Google <00:08:42.385>search

207
00:08:44.265 --> 00:08:47.485
<00:08:44.265>and <00:08:44.392>the <00:08:44.519>users <00:08:44.646>just <00:08:44.774>says <00:08:44.901>hi, <00:08:45.028>then <00:08:45.155>we <00:08:45.283>of <00:08:45.410>course <00:08:45.537>don't <00:08:45.665>want

208
00:08:47.485 --> 00:08:48.485
<00:08:47.485>to <00:08:47.710>use <00:08:47.935>the <00:08:48.160>search <00:08:48.385>tool.

209
00:08:48.745 --> 00:08:51.325
<00:08:48.745>We <00:08:48.900>want <00:08:49.056>to <00:08:49.211>respond <00:08:49.367>with <00:08:49.522>a <00:08:49.678>very <00:08:49.833>nice, <00:08:49.989>very <00:08:50.145>like,

210
00:08:51.345 --> 00:08:52.445
<00:08:51.345>hi, <00:08:51.543>how <00:08:51.741>can <00:08:51.939>I <00:08:52.137>help <00:08:52.335>you?

211
00:08:52.705 --> 00:08:54.325
<00:08:52.705>But <00:08:52.905>if <00:08:53.105>the <00:08:53.305>user <00:08:53.505>wants <00:08:53.705>to <00:08:53.905>ask, <00:08:54.105>okay,

212
00:08:54.325 --> 00:08:55.525
<00:08:54.325>like <00:08:54.541>what's <00:08:54.757>the <00:08:54.973>capital <00:08:55.189>of <00:08:55.405>Germany?

213
00:08:56.025 --> 00:08:58.685
<00:08:56.025>Um, <00:08:56.180>that <00:08:56.336>might <00:08:56.491>be <00:08:56.647>some <00:08:56.802>question <00:08:56.958>where <00:08:57.113>we <00:08:57.269>would <00:08:57.425>like

214
00:08:58.685 --> 00:08:59.925
<00:08:58.685>to <00:08:58.908>use <00:08:59.131>the <00:08:59.354>Google <00:08:59.577>search <00:08:59.800>tool.

215
00:09:00.705 --> 00:09:03.885
<00:09:00.705>Uh, <00:09:00.832>very <00:09:00.959>cool. <00:09:01.086>Um, <00:09:01.214>you <00:09:01.341>can <00:09:01.468>use <00:09:01.595>it <00:09:01.723>to <00:09:01.850>basically <00:09:01.977>connect <00:09:02.105>the

216
00:09:03.885 --> 00:09:07.085
<00:09:03.885>outside <00:09:04.025>world <00:09:04.165>with <00:09:04.305>the <00:09:04.444>LLM <00:09:04.585>with <00:09:04.725>a <00:09:04.865>much <00:09:05.005>nicer <00:09:05.145>user <00:09:05.285>interface,

217
00:09:07.465 --> 00:09:08.685
<00:09:07.465>um, <00:09:07.739>than <00:09:08.014>we <00:09:08.288>had <00:09:08.563>before.

218
00:09:09.595 --> 00:09:11.645
<00:09:09.595>Then <00:09:09.795>we <00:09:09.995>have <00:09:10.195>the, <00:09:10.395>the <00:09:10.595>planning <00:09:10.795>and <00:09:10.995>uh,

219
00:09:11.745 --> 00:09:13.045
<00:09:11.745>or <00:09:12.329>orchestration <00:09:12.915>pattern.

220
00:09:13.425 --> 00:09:14.525
<00:09:13.425>That's <00:09:13.920>basically <00:09:14.415>where

221
00:09:14.665 --> 00:09:17.885
<00:09:14.665>before <00:09:14.944>executing <00:09:15.224>any <00:09:15.505>tasks, <00:09:15.785>any <00:09:16.064>generation,

222
00:09:17.885 --> 00:09:20.965
<00:09:17.885>we <00:09:18.060>have <00:09:18.235>some <00:09:18.410>kind <00:09:18.585>of <00:09:18.760>LLM <00:09:18.935>planner <00:09:19.110>which <00:09:19.285>tries

223
00:09:21.025 --> 00:09:24.565
<00:09:21.025>to <00:09:21.199>break <00:09:21.375>down <00:09:21.550>the <00:09:21.725>US <00:09:21.900>vary <00:09:22.074>into <00:09:22.250>multiple <00:09:22.425>subtasks,

224
00:09:24.785 --> 00:09:28.365
<00:09:24.785>and <00:09:24.959>then <00:09:25.135>those <00:09:25.310>subtasks <00:09:25.485>are <00:09:25.660>forwarded <00:09:25.834>to <00:09:26.010>different <00:09:26.185>workers.

225
00:09:28.545 --> 00:09:31.965
<00:09:28.545>And <00:09:28.685>those <00:09:28.824>workers <00:09:28.964>can <00:09:29.104>be <00:09:29.245>normal <00:09:29.385>LLM <00:09:29.525>queries <00:09:29.665>like <00:09:29.805>in <00:09:29.944>the

226
00:09:31.965 --> 00:09:34.685
<00:09:31.965>reflection <00:09:32.140>pattern <00:09:32.315>or <00:09:32.490>it <00:09:32.665>can <00:09:32.840>be <00:09:33.015>tool <00:09:33.190>use <00:09:33.365>agent.

227
00:09:35.065 --> 00:09:37.645
<00:09:35.065>Um, <00:09:35.220>I <00:09:35.376>think <00:09:35.531>a <00:09:35.687>good <00:09:35.842>example <00:09:35.998>of <00:09:36.153>like <00:09:36.309>this <00:09:36.465>kind

228
00:09:37.645 --> 00:09:41.285
<00:09:37.645>of <00:09:37.819>agent <00:09:37.995>is <00:09:38.170>now <00:09:38.345>like <00:09:38.520>GitHub <00:09:38.694>copilot <00:09:38.870>or <00:09:39.045>cursor.

229
00:09:41.285 --> 00:09:44.405
<00:09:41.285>All <00:09:41.412>of <00:09:41.539>those <00:09:41.666>use <00:09:41.794>some <00:09:41.921>kind <00:09:42.048>of, <00:09:42.175>um, <00:09:42.303>LLM <00:09:42.430>or <00:09:42.557>planning <00:09:42.685>pattern

230
00:09:44.425 --> 00:09:47.325
<00:09:44.425>or <00:09:44.580>even <00:09:44.736>like <00:09:44.891>deep <00:09:45.047>research <00:09:45.202>agents <00:09:45.358>where <00:09:45.513>we <00:09:45.669>oftentimes <00:09:45.824>have,

231
00:09:47.345 --> 00:09:51.165
<00:09:47.345>Hey, <00:09:47.461>I <00:09:47.578>would <00:09:47.695>like <00:09:47.811>to <00:09:47.928>research <00:09:48.045>what <00:09:48.161>are <00:09:48.278>like <00:09:48.395>the <00:09:48.511>top <00:09:48.628>25, <00:09:48.745>uh,

232
00:09:51.165 --> 00:09:52.965
<00:09:51.165>innovations <00:09:51.515>in, <00:09:51.865>uh, <00:09:52.214>renewable <00:09:52.564>energy.

233
00:09:53.115 --> 00:09:55.845
<00:09:53.115>Then <00:09:53.255>the <00:09:53.395>LLM <00:09:53.535>planner <00:09:53.675>would <00:09:53.815>create <00:09:53.955>some <00:09:54.095>kind <00:09:54.235>of <00:09:54.375>plane, <00:09:54.515>okay,

234
00:09:55.845 --> 00:09:57.125
<00:09:55.845>what <00:09:56.075>would <00:09:56.305>I <00:09:56.536>need <00:09:56.766>to <00:09:56.997>search?

235
00:09:57.585 --> 00:09:59.725
<00:09:57.585>Uh, <00:09:57.865>and <00:09:58.145>then <00:09:58.425>different <00:09:58.705>workers <00:09:58.985>start

236
00:09:59.725 --> 00:10:01.045
<00:09:59.725>searching <00:10:00.121>those <00:10:00.516>different <00:10:00.913>topics.

237
00:10:01.475 --> 00:10:03.205
<00:10:01.475>Then <00:10:01.675>I <00:10:01.875>get <00:10:02.075>back <00:10:02.275>all <00:10:02.475>of <00:10:02.675>those <00:10:02.875>results.

238
00:10:03.585 --> 00:10:05.085
<00:10:03.585>And <00:10:03.810>then <00:10:04.035>I <00:10:04.260>might <00:10:04.485>have <00:10:04.710>another <00:10:04.935>LLM

239
00:10:05.085 --> 00:10:08.645
<00:10:05.085>or <00:10:05.260>different <00:10:05.435>agents <00:10:05.610>which <00:10:05.785>continue, <00:10:05.960>um, <00:10:06.135>on <00:10:06.310>the <00:10:06.485>report.

240
00:10:10.145 --> 00:10:13.525
<00:10:10.145>And <00:10:10.378>then, <00:10:10.611>uh, <00:10:10.845>more <00:10:11.078>now <00:10:11.311>trending <00:10:11.545>topic

241
00:10:13.625 --> 00:10:16.805
<00:10:13.625>or <00:10:13.800>like <00:10:13.975>pattern <00:10:14.150>is <00:10:14.325>the <00:10:14.500>multi-agent <00:10:14.675>pattern <00:10:14.850>where <00:10:15.025>instead

242
00:10:16.805 --> 00:10:19.685
<00:10:16.805>of <00:10:16.960>having <00:10:17.116>a <00:10:17.271>planner <00:10:17.427>in <00:10:17.582>front, <00:10:17.738>we <00:10:17.893>have <00:10:18.049>multiple <00:10:18.204>agents

243
00:10:20.155 --> 00:10:23.325
<00:10:20.155>with <00:10:20.435>very <00:10:20.714>specific <00:10:20.995>personas <00:10:21.275>and <00:10:21.555>features

244
00:10:23.545 --> 00:10:24.765
<00:10:23.545>and <00:10:24.093>they <00:10:24.643>collaborate

245
00:10:24.765 --> 00:10:27.245
<00:10:24.765>with <00:10:24.965>each <00:10:25.165>other <00:10:25.365>based <00:10:25.564>on <00:10:25.765>a <00:10:25.965>user's <00:10:26.165>request.

246
00:10:27.545 --> 00:10:29.565
<00:10:27.545>So <00:10:27.778>if <00:10:28.011>I <00:10:28.245>have <00:10:28.478>a <00:10:28.711>user <00:10:28.944>input

247
00:10:29.585 --> 00:10:32.445
<00:10:29.585>and <00:10:29.712>I <00:10:29.839>have <00:10:29.966>one <00:10:30.094>agent, <00:10:30.221>which <00:10:30.348>is <00:10:30.475>a <00:10:30.603>hotel <00:10:30.730>book <00:10:30.857>booking <00:10:30.985>agent

248
00:10:32.705 --> 00:10:35.245
<00:10:32.705>and <00:10:32.880>another <00:10:33.055>agent <00:10:33.230>which <00:10:33.405>is <00:10:33.580>a <00:10:33.755>flight <00:10:33.930>booking <00:10:34.105>agent,

249
00:10:35.345 --> 00:10:38.085
<00:10:35.345>so <00:10:35.545>I <00:10:35.745>have <00:10:35.945>two <00:10:36.145>very <00:10:36.345>separate <00:10:36.545>specific <00:10:36.745>agents,

250
00:10:38.515 --> 00:10:39.925
<00:10:38.515>very <00:10:39.149>specific <00:10:39.784>instruction.

251
00:10:40.305 --> 00:10:43.205
<00:10:40.305>And <00:10:40.460>when <00:10:40.616>building, <00:10:40.771>uh, <00:10:40.927>a <00:10:41.082>system, <00:10:41.238>I <00:10:41.393>can <00:10:41.549>combine <00:10:41.704>them

252
00:10:43.425 --> 00:10:46.405
<00:10:43.425>for <00:10:43.580>user <00:10:43.736>if <00:10:43.891>they <00:10:44.047>want <00:10:44.202>to <00:10:44.358>like <00:10:44.513>book <00:10:44.669>a <00:10:44.824>flight

253
00:10:46.465 --> 00:10:48.405
<00:10:46.465>and <00:10:46.665>uh, <00:10:46.865>a <00:10:47.065>hotel <00:10:47.265>at <00:10:47.465>the <00:10:47.665>same <00:10:47.865>time

254
00:10:48.705 --> 00:10:50.245
<00:10:48.705>or <00:10:48.903>can <00:10:49.101>I <00:10:49.299>can <00:10:49.497>have <00:10:49.695>them <00:10:49.893>like <00:10:50.091>separate.

255
00:10:50.305 --> 00:10:53.325
<00:10:50.305>The <00:10:50.505>benefit <00:10:50.704>of <00:10:50.905>building <00:10:51.104>multi-agent, <00:10:51.305>uh, <00:10:51.505>system <00:10:51.704>is

256
00:10:53.325 --> 00:10:55.645
<00:10:53.325>that <00:10:53.480>you <00:10:53.636>can <00:10:53.791>like <00:10:53.947>test <00:10:54.102>them <00:10:54.258>and <00:10:54.413>debug <00:10:54.569>them <00:10:54.725>individually.

257
00:10:55.875 --> 00:10:57.565
<00:10:55.875>It's <00:10:56.108>so <00:10:56.341>that <00:10:56.575>you're <00:10:56.808>building <00:10:57.041>one <00:10:57.275>agent

258
00:10:57.795 --> 00:11:00.925
<00:10:57.795>with <00:10:57.969>like <00:10:58.145>tools <00:10:58.319>which <00:10:58.495>have <00:10:58.670>access <00:10:58.844>to <00:10:59.020>booking <00:10:59.194>flights

259
00:11:01.105 --> 00:11:04.205
<00:11:01.105>and <00:11:01.260>booking <00:11:01.416>hotels <00:11:01.571>at <00:11:01.727>the <00:11:01.882>same <00:11:02.038>time <00:11:02.193>you <00:11:02.349>separate <00:11:02.505>them.

260
00:11:04.345 --> 00:11:06.125
<00:11:04.345>And <00:11:04.695>similar <00:11:05.045>to <00:11:05.395>software <00:11:05.745>development

261
00:11:06.125 --> 00:11:07.645
<00:11:06.125>where <00:11:06.398>we <00:11:06.672>are <00:11:06.945>now <00:11:07.219>building <00:11:07.492>microservices,

262
00:11:08.035 --> 00:11:10.205
<00:11:08.035>it's <00:11:08.235>much <00:11:08.435>much <00:11:08.635>easier <00:11:08.834>for <00:11:09.035>us <00:11:09.235>to <00:11:09.435>identify

263
00:11:10.205 --> 00:11:11.245
<00:11:10.205>where <00:11:10.517>our <00:11:10.829>issues <00:11:11.141>are.

264
00:11:11.465 --> 00:11:13.005
<00:11:11.465>We <00:11:11.696>can <00:11:11.927>test <00:11:12.158>them <00:11:12.389>and <00:11:12.620>evaluate <00:11:12.851>them

265
00:11:13.665 --> 00:11:17.245
<00:11:13.665>and <00:11:13.865>yeah, <00:11:14.064>so <00:11:14.265>why <00:11:14.464>do <00:11:14.665>their <00:11:14.865>patterns <00:11:15.064>matter?

266
00:11:17.745 --> 00:11:20.165
<00:11:17.745>Um, <00:11:17.945>I <00:11:18.145>think <00:11:18.345>it's <00:11:18.545>very <00:11:18.745>important <00:11:18.945>that, <00:11:19.145>uh,

267
00:11:20.165 --> 00:11:22.925
<00:11:20.165>for <00:11:20.339>us <00:11:20.515>pattern <00:11:20.689>provide <00:11:20.865>us <00:11:21.040>a <00:11:21.214>structured <00:11:21.390>way <00:11:21.564>of

268
00:11:22.945 --> 00:11:25.045
<00:11:22.945>how <00:11:23.225>we <00:11:23.505>can <00:11:23.785>approach <00:11:24.065>building <00:11:24.345>agents

269
00:11:25.105 --> 00:11:28.525
<00:11:25.105>and <00:11:25.245>how <00:11:25.385>we <00:11:25.525>can <00:11:25.665>solve <00:11:25.805>them <00:11:25.945>by <00:11:26.085>having <00:11:26.225>those <00:11:26.365>design <00:11:26.505>principles.

270
00:11:28.805 --> 00:11:30.765
<00:11:28.805>Patterns <00:11:29.038>as <00:11:29.271>mentioned, <00:11:29.505>are <00:11:29.738>not <00:11:29.971>fixed <00:11:30.204>rules.

271
00:11:30.865 --> 00:11:31.885
<00:11:30.865>So <00:11:31.171>they're <00:11:31.477>building <00:11:31.783>blocks.

272
00:11:31.905 --> 00:11:34.485
<00:11:31.905>So <00:11:32.138>you <00:11:32.371>can <00:11:32.605>combine <00:11:32.838>a <00:11:33.071>orchestration <00:11:33.305>pattern

273
00:11:34.515 --> 00:11:37.725
<00:11:34.515>with <00:11:34.655>a <00:11:34.795>tool <00:11:34.935>use <00:11:35.074>pattern <00:11:35.215>or <00:11:35.355>multi-agent <00:11:35.495>pattern <00:11:35.635>with <00:11:35.775>tool <00:11:35.915>use

274
00:11:37.785 --> 00:11:39.045
<00:11:37.785>or <00:11:38.163>everything <00:11:38.540>at <00:11:38.919>once.

275
00:11:39.145 --> 00:11:40.445
<00:11:39.145>You <00:11:39.340>will <00:11:39.535>see <00:11:39.730>when <00:11:39.925>you <00:11:40.120>build <00:11:40.315>agents

276
00:11:40.445 --> 00:11:42.845
<00:11:40.445>that <00:11:40.725>those <00:11:41.005>are <00:11:41.285>more <00:11:41.565>like <00:11:41.845>guidelines

277
00:11:42.865 --> 00:11:44.405
<00:11:42.865>and <00:11:43.142>help <00:11:43.419>you <00:11:43.696>think <00:11:43.973>about, <00:11:44.251>okay,

278
00:11:44.405 --> 00:11:46.965
<00:11:44.405>how <00:11:44.560>can <00:11:44.716>I <00:11:44.871>solve <00:11:45.027>certain <00:11:45.182>problems <00:11:45.338>in <00:11:45.493>my <00:11:45.649>use <00:11:45.805>case?

279
00:11:47.745 --> 00:11:50.525
<00:11:47.745>And <00:11:47.900>of <00:11:48.056>course <00:11:48.211>as <00:11:48.367>mentioned, <00:11:48.522>they <00:11:48.678>help <00:11:48.833>us <00:11:48.989>really <00:11:49.145>try

280
00:11:50.525 --> 00:11:53.725
<00:11:50.525>to <00:11:50.725>scale <00:11:50.925>our <00:11:51.125>application <00:11:51.324>because <00:11:51.525>the <00:11:51.725>more <00:11:51.925>complex

281
00:11:53.825 --> 00:11:56.765
<00:11:53.825>and <00:11:53.980>the <00:11:54.136>bigger <00:11:54.291>agents <00:11:54.447>get, <00:11:54.602>we <00:11:54.758>really <00:11:54.913>need <00:11:55.069>clear <00:11:55.225>lines

282
00:11:56.765 --> 00:11:58.685
<00:11:56.765>between <00:11:57.045>what <00:11:57.324>an <00:11:57.605>agent <00:11:57.885>is <00:11:58.165>doing

283
00:11:58.705 --> 00:12:00.405
<00:11:58.705>and <00:11:58.938>especially <00:11:59.171>how <00:11:59.405>can <00:11:59.638>I <00:11:59.871>test <00:12:00.105>them?

284
00:12:00.405 --> 00:12:02.925
<00:12:00.405>Because <00:12:00.579>if <00:12:00.755>you <00:12:00.930>have <00:12:01.105>like <00:12:01.280>very <00:12:01.454>complex <00:12:01.630>multi-agent <00:12:01.805>systems

285
00:12:03.745 --> 00:12:07.445
<00:12:03.745>and <00:12:03.900>something <00:12:04.056>fails, <00:12:04.211>it <00:12:04.367>is <00:12:04.522>very <00:12:04.678>hard <00:12:04.833>currently <00:12:04.989>to <00:12:05.145>understand

286
00:12:07.445 --> 00:12:10.005
<00:12:07.445>where <00:12:07.645>it <00:12:07.845>failed <00:12:08.045>because <00:12:08.245>very <00:12:08.445>different <00:12:08.645>to <00:12:08.845>software

287
00:12:10.005 --> 00:12:11.605
<00:12:10.005>where <00:12:10.238>we <00:12:10.471>have <00:12:10.705>all <00:12:10.938>of <00:12:11.171>those <00:12:11.405>tracing

288
00:12:12.195 --> 00:12:13.565
<00:12:12.195>LLMs <00:12:12.606>are <00:12:13.017>not <00:12:13.428>deterministic.

289
00:12:13.665 --> 00:12:15.605
<00:12:13.665>So <00:12:13.865>we <00:12:14.064>really <00:12:14.265>need <00:12:14.464>to <00:12:14.665>know <00:12:14.865>each <00:12:15.064>step

290
00:12:15.985 --> 00:12:19.405
<00:12:15.985>and <00:12:16.125>what <00:12:16.265>could <00:12:16.405>be <00:12:16.545>the <00:12:16.685>result <00:12:16.825>based <00:12:16.965>on <00:12:17.105>the <00:12:17.245>user's <00:12:17.385>input.

291
00:12:19.545 --> 00:12:22.005
<00:12:19.545>And <00:12:19.719>with <00:12:19.895>those <00:12:20.069>patterns, <00:12:20.245>we <00:12:20.420>have <00:12:20.594>much <00:12:20.770>clearer <00:12:20.944>reliance

292
00:12:22.005 --> 00:12:24.005
<00:12:22.005>between, <00:12:22.238>okay, <00:12:22.471>what <00:12:22.705>are <00:12:22.938>the <00:12:23.171>each, <00:12:23.405>um,

293
00:12:24.155 --> 00:12:25.285
<00:12:24.155>each <00:12:24.663>section's <00:12:25.172>handling?

294
00:12:27.275 --> 00:12:30.685
<00:12:27.275>Awesome. <00:12:27.430>So <00:12:27.586>how <00:12:27.741>are <00:12:27.897>we <00:12:28.052>going <00:12:28.208>to <00:12:28.363>evaluate <00:12:28.519>those <00:12:28.675>agents?

295
00:12:31.345 --> 00:12:35.045
<00:12:31.345>And <00:12:31.500>maybe <00:12:31.656>you <00:12:31.811>have <00:12:31.967>trained <00:12:32.122>models, <00:12:32.278>maybe <00:12:32.433>use <00:12:32.589>LLMs <00:12:32.745>already.

296
00:12:35.455 --> 00:12:38.405
<00:12:35.455>Agent <00:12:35.630>evaluation <00:12:35.805>is <00:12:35.980>a <00:12:36.155>bit <00:12:36.330>different, <00:12:36.505>I <00:12:36.680>would <00:12:36.855>say.

297
00:12:38.725 --> 00:12:40.245
<00:12:38.725>I <00:12:38.920>try <00:12:39.115>to <00:12:39.311>always <00:12:39.506>compare <00:12:39.702>it <00:12:39.897>to <00:12:40.093>like

298
00:12:40.245 --> 00:12:42.765
<00:12:40.245>what <00:12:40.420>we <00:12:40.595>have <00:12:40.770>in <00:12:40.945>LLM <00:12:41.120>evaluation <00:12:41.295>and <00:12:41.470>agent <00:12:41.645>evaluation.

299
00:12:42.865 --> 00:12:44.925
<00:12:42.865>In <00:12:43.098>LLM <00:12:43.331>evaluation, <00:12:43.565>we <00:12:43.798>have <00:12:44.031>an <00:12:44.265>input

300
00:12:44.925 --> 00:12:46.885
<00:12:44.925>and <00:12:45.158>an <00:12:45.391>output, <00:12:45.625>very <00:12:45.858>easy <00:12:46.091>to <00:12:46.324>evaluate.

301
00:12:46.965 --> 00:12:50.165
<00:12:46.965>I <00:12:47.092>have <00:12:47.219>a <00:12:47.346>task <00:12:47.474>with, <00:12:47.601>okay, <00:12:47.728>write <00:12:47.855>a <00:12:47.983>nice <00:12:48.110>essay <00:12:48.237>about <00:12:48.365>Berlin.

302
00:12:50.525 --> 00:12:52.605
<00:12:50.525>I <00:12:50.699>have <00:12:50.875>an <00:12:51.050>output, <00:12:51.225>I <00:12:51.400>can <00:12:51.574>evaluate <00:12:51.750>that <00:12:51.925>output.

303
00:12:53.105 --> 00:12:54.605
<00:12:53.105>In <00:12:53.375>agents <00:12:53.645>it's <00:12:53.915>a <00:12:54.185>bit <00:12:54.455>different

304
00:12:54.605 --> 00:12:57.645
<00:12:54.605>because <00:12:54.760>normally <00:12:54.916>I <00:12:55.071>want <00:12:55.227>an <00:12:55.382>agent <00:12:55.538>to <00:12:55.693>take <00:12:55.849>an <00:12:56.005>action.

305
00:12:57.905 --> 00:13:00.685
<00:12:57.905>So <00:12:58.079>how <00:12:58.255>do <00:12:58.430>I <00:12:58.605>evaluate <00:12:58.780>if <00:12:58.954>the <00:12:59.130>Google <00:12:59.305>search

306
00:13:00.785 --> 00:13:02.165
<00:13:00.785>was <00:13:01.095>executed <00:13:01.406>correctly <00:13:01.716>or <00:13:02.026>not?

307
00:13:02.305 --> 00:13:04.725
<00:13:02.305>So <00:13:02.444>I <00:13:02.584>need <00:13:02.724>have <00:13:02.864>an <00:13:03.005>additional <00:13:03.145>step <00:13:03.285>to <00:13:03.425>make <00:13:03.564>sure, <00:13:03.704>okay,

308
00:13:04.725 --> 00:13:07.085
<00:13:04.725>which <00:13:05.191>queries <00:13:05.658>should <00:13:06.125>search,

309
00:13:07.135 --> 00:13:10.765
<00:13:07.135>which <00:13:07.368>query <00:13:07.601>should <00:13:07.835>not <00:13:08.068>search <00:13:08.301>then <00:13:08.535>cost.

310
00:13:11.065 --> 00:13:15.285
<00:13:11.065>And <00:13:11.240>normal <00:13:11.415>MLLM <00:13:11.590>evaluation <00:13:11.765>is <00:13:11.940>very <00:13:12.115>easy <00:13:12.290>to <00:13:12.465>calculate.

311
00:13:15.445 --> 00:13:18.005
<00:13:15.445>I <00:13:15.572>know <00:13:15.699>my <00:13:15.826>input, <00:13:15.954>I <00:13:16.081>know <00:13:16.208>my <00:13:16.335>output, <00:13:16.463>I <00:13:16.590>know <00:13:16.717>the <00:13:16.845>cost.

312
00:13:18.395 --> 00:13:21.165
<00:13:18.395>With, <00:13:18.550>uh, <00:13:18.706>agents <00:13:18.861>we <00:13:19.017>learned <00:13:19.172>they <00:13:19.328>run <00:13:19.483>in <00:13:19.639>a <00:13:19.795>loop.

313
00:13:21.345 --> 00:13:25.005
<00:13:21.345>So <00:13:21.485>I <00:13:21.625>don't <00:13:21.765>know <00:13:21.905>always <00:13:22.045>how <00:13:22.185>many <00:13:22.325>loops <00:13:22.465>it <00:13:22.605>will <00:13:22.745>take

314
00:13:25.145 --> 00:13:26.925
<00:13:25.145>for <00:13:25.378>an <00:13:25.611>agent <00:13:25.845>to <00:13:26.078>complete <00:13:26.311>its <00:13:26.545>task.

315
00:13:27.305 --> 00:13:29.405
<00:13:27.305>So <00:13:27.505>I <00:13:27.704>only <00:13:27.905>have <00:13:28.104>a <00:13:28.305>range <00:13:28.505>of <00:13:28.704>price

316
00:13:29.705 --> 00:13:31.085
<00:13:29.705>and <00:13:29.882>there <00:13:30.059>I <00:13:30.237>really <00:13:30.414>need <00:13:30.592>to <00:13:30.769>make <00:13:30.947>sure

317
00:13:31.085 --> 00:13:32.885
<00:13:31.085>that <00:13:31.285>I <00:13:31.485>have <00:13:31.685>some <00:13:31.885>guardrails <00:13:32.085>in <00:13:32.285>place, <00:13:32.485>not

318
00:13:32.885 --> 00:13:35.045
<00:13:32.885>that <00:13:33.165>my <00:13:33.444>agent <00:13:33.725>basically <00:13:34.005>loops <00:13:34.285>infinitely

319
00:13:35.185 --> 00:13:36.805
<00:13:35.185>and <00:13:35.359>I <00:13:35.535>end <00:13:35.709>up <00:13:35.885>with <00:13:36.060>a <00:13:36.234>very <00:13:36.410>big <00:13:36.584>bill.

320
00:13:37.345 --> 00:13:40.485
<00:13:37.345>Um, <00:13:37.578>then <00:13:37.811>LLM <00:13:38.045>evaluation <00:13:38.278>is <00:13:38.511>normally <00:13:38.745>done,

321
00:13:40.545 --> 00:13:44.045
<00:13:40.545>or <00:13:40.778>we <00:13:41.011>have <00:13:41.245>existing <00:13:41.478>benchmarks <00:13:41.711>every <00:13:41.944>time

322
00:13:44.585 --> 00:13:47.805
<00:13:44.585>we <00:13:44.760>open <00:13:44.935>ai, <00:13:45.110>andro <00:13:45.285>pick <00:13:45.460>releases <00:13:45.635>a <00:13:45.810>new <00:13:45.985>model.

323
00:13:47.815 --> 00:13:49.325
<00:13:47.815>There <00:13:48.268>are <00:13:48.721>academic <00:13:49.174>benchmarks

324
00:13:49.325 --> 00:13:50.845
<00:13:49.325>and <00:13:49.598>benchmarks <00:13:49.872>to <00:13:50.145>show <00:13:50.419>the <00:13:50.693>performance.

325
00:13:51.185 --> 00:13:53.085
<00:13:51.185>For <00:13:51.535>agents, <00:13:51.885>it's <00:13:52.234>very <00:13:52.584>different.

326
00:13:53.405 --> 00:13:54.405
<00:13:53.405>Normally <00:13:53.704>we <00:13:54.005>build <00:13:54.305>agents

327
00:13:54.465 --> 00:13:57.165
<00:13:54.465>to <00:13:54.620>solve <00:13:54.776>very <00:13:54.931>specific <00:13:55.087>ca <00:13:55.242>tasks <00:13:55.398>in <00:13:55.553>the <00:13:55.709>real <00:13:55.865>world.

328
00:13:57.665 --> 00:14:02.085
<00:13:57.665>So <00:13:57.839>a <00:13:58.015>hotel <00:13:58.189>booking <00:13:58.365>agent <00:13:58.540>for <00:13:58.714>booking, <00:13:58.890>don't <00:13:59.064>booking.com,

329
00:14:02.375 --> 00:14:03.965
<00:14:02.375>there <00:14:02.608>is <00:14:02.841>no <00:14:03.075>benchmark <00:14:03.308>for <00:14:03.541>it, <00:14:03.775>right?

330
00:14:04.265 --> 00:14:05.885
<00:14:04.265>So <00:14:04.545>you <00:14:04.824>need <00:14:05.105>to <00:14:05.385>invest <00:14:05.665>time

331
00:14:06.265 --> 00:14:09.685
<00:14:06.265>and <00:14:06.405>resources <00:14:06.545>to <00:14:06.685>make <00:14:06.824>sure <00:14:06.965>that <00:14:07.105>you <00:14:07.245>can <00:14:07.385>evaluate <00:14:07.525>your <00:14:07.665>agent.

332
00:14:09.785 --> 00:14:12.805
<00:14:09.785>And <00:14:09.940>with <00:14:10.096>an <00:14:10.251>LLM <00:14:10.407>you <00:14:10.562>can <00:14:10.718>easily <00:14:10.873>compare <00:14:11.029>between <00:14:11.185>tropic

333
00:14:12.805 --> 00:14:15.165
<00:14:12.805>and <00:14:13.084>OpenAI, <00:14:13.364>which <00:14:13.645>model <00:14:13.925>might <00:14:14.204>be

334
00:14:15.165 --> 00:14:16.365
<00:14:15.165>better <00:14:15.435>on <00:14:15.704>math, <00:14:15.975>for <00:14:16.245>example.

335
00:14:17.225 --> 00:14:18.725
<00:14:17.225>And <00:14:17.562>then <00:14:17.900>of <00:14:18.237>course, <00:14:18.575>uh,

336
00:14:18.845 --> 00:14:22.645
<00:14:18.845>LLM <00:14:19.125>evaluation <00:14:19.405>is <00:14:19.685>mostly <00:14:19.965>measuring <00:14:20.245>capabilities

337
00:14:23.385 --> 00:14:25.965
<00:14:23.385>and <00:14:23.585>with <00:14:23.785>agents <00:14:23.985>we <00:14:24.185>need <00:14:24.385>to <00:14:24.585>measure <00:14:24.785>reliability

338
00:14:26.935 --> 00:14:29.405
<00:14:26.935>since <00:14:27.401>capabilities <00:14:27.868>defines <00:14:28.334>okay,

339
00:14:29.785 --> 00:14:33.045
<00:14:29.785>how <00:14:29.940>is <00:14:30.096>the <00:14:30.251>the <00:14:30.407>the <00:14:30.562>success <00:14:30.718>rate <00:14:30.873>of <00:14:31.029>my <00:14:31.185>model?

340
00:14:33.105 --> 00:14:36.165
<00:14:33.105>So <00:14:33.245>if <00:14:33.385>I <00:14:33.525>ask <00:14:33.665>it, <00:14:33.805>uh, <00:14:33.945>one <00:14:34.085>math <00:14:34.225>questions <00:14:34.365>10 <00:14:34.505>times,

341
00:14:36.825 --> 00:14:38.045
<00:14:36.825>did <00:14:37.008>it <00:14:37.191>get <00:14:37.374>it <00:14:37.557>right <00:14:37.740>or <00:14:37.923>not?

342
00:14:38.345 --> 00:14:41.405
<00:14:38.345>And <00:14:38.500>that's <00:14:38.656>what, <00:14:38.811>uh, <00:14:38.967>researchers <00:14:39.122>most <00:14:39.278>of <00:14:39.433>the <00:14:39.589>time <00:14:39.745>report.

343
00:14:41.705 --> 00:14:44.725
<00:14:41.705>And <00:14:41.832>for <00:14:41.959>agents, <00:14:42.086>we <00:14:42.214>really <00:14:42.341>need <00:14:42.468>to <00:14:42.595>make <00:14:42.723>sure <00:14:42.850>that <00:14:42.977>our <00:14:43.105>agent,

344
00:14:44.725 --> 00:14:45.925
<00:14:44.725>when <00:14:44.941>I <00:14:45.157>run <00:14:45.372>it <00:14:45.589>10 <00:14:45.805>times,

345
00:14:46.675 --> 00:14:50.325
<00:14:46.675>that <00:14:46.830>it <00:14:46.986>is <00:14:47.141>successful <00:14:47.297>nine <00:14:47.452>times <00:14:47.608>at <00:14:47.763>a <00:14:47.919>time, <00:14:48.074>right?

346
00:14:50.545 --> 00:14:52.685
<00:14:50.545>And <00:14:50.685>if <00:14:50.824>I'm <00:14:50.964>running <00:14:51.104>an <00:14:51.245>agent <00:14:51.385>one <00:14:51.525>time <00:14:51.665>and <00:14:51.805>it <00:14:51.944>works,

347
00:14:53.145 --> 00:14:55.325
<00:14:53.145>but <00:14:53.285>the <00:14:53.425>second <00:14:53.564>time <00:14:53.704>it <00:14:53.845>fails, <00:14:53.985>the <00:14:54.125>third <00:14:54.265>time <00:14:54.405>it <00:14:54.545>fails

348
00:14:55.325 --> 00:14:57.405
<00:14:55.325>and <00:14:55.465>the <00:14:55.605>fifth <00:14:55.745>time <00:14:55.885>it <00:14:56.025>fails, <00:14:56.165>it's <00:14:56.305>a <00:14:56.445>very <00:14:56.585>bad <00:14:56.725>agent.

349
00:14:57.805 --> 00:15:01.405
<00:14:57.805>I <00:14:57.932>try <00:14:58.059>to <00:14:58.186>visualize <00:14:58.314>this <00:14:58.441>if <00:14:58.568>a, <00:14:58.695>with <00:14:58.823>a <00:14:58.950>very <00:14:59.077>simple <00:14:59.204>example,

350
00:15:01.825 --> 00:15:05.685
<00:15:01.825>and <00:15:01.932>if <00:15:02.040>we <00:15:02.148>look <00:15:02.255>at <00:15:02.363>the <00:15:02.471>past <00:15:02.578>one <00:15:02.686>rate, <00:15:02.794>meaning <00:15:02.901>I <00:15:03.009>run <00:15:03.117>one <00:15:03.225>task

351
00:15:05.685 --> 00:15:08.765
<00:15:05.685>with <00:15:05.885>agent <00:15:06.084>one <00:15:06.285>and <00:15:06.484>agent <00:15:06.685>two <00:15:06.885>during <00:15:07.084>evaluation,

352
00:15:09.295 --> 00:15:12.405
<00:15:09.295>agent <00:15:09.469>one <00:15:09.645>has <00:15:09.819>a <00:15:09.995>60% <00:15:10.170>success <00:15:10.344>rate <00:15:10.520>agent <00:15:10.694>two,

353
00:15:12.725 --> 00:15:14.085
<00:15:12.725>a <00:15:13.031>little <00:15:13.337>bit <00:15:13.643>below <00:15:13.949>50%.

354
00:15:14.435 --> 00:15:17.685
<00:15:14.435>Very <00:15:14.590>naively <00:15:14.746>speaking, <00:15:14.901>all <00:15:15.057>of <00:15:15.212>us <00:15:15.368>would <00:15:15.523>probably <00:15:15.679>say, <00:15:15.834>okay,

355
00:15:18.135 --> 00:15:20.085
<00:15:18.135>let's <00:15:18.415>use <00:15:18.694>agent <00:15:18.975>one, <00:15:19.255>it's <00:15:19.535>better.

356
00:15:20.785 --> 00:15:22.845
<00:15:20.785>But <00:15:21.018>if <00:15:21.251>we <00:15:21.485>look <00:15:21.718>at, <00:15:21.951>uh, <00:15:22.185>reliability,

357
00:15:22.845 --> 00:15:26.125
<00:15:22.845>meaning <00:15:22.985>I <00:15:23.125>run <00:15:23.265>the <00:15:23.405>same <00:15:23.545>prompt, <00:15:23.685>same <00:15:23.825>task <00:15:23.965>for <00:15:24.105>my <00:15:24.245>user

358
00:15:26.965 --> 00:15:30.045
<00:15:26.965>multiple <00:15:27.165>times <00:15:27.365>we <00:15:27.565>see <00:15:27.765>agent <00:15:27.965>two <00:15:28.165>perform <00:15:28.365>better

359
00:15:30.905 --> 00:15:33.005
<00:15:30.905>or <00:15:31.138>is <00:15:31.371>more <00:15:31.605>reliable <00:15:31.838>than <00:15:32.071>agent <00:15:32.305>one.

360
00:15:33.345 --> 00:15:36.525
<00:15:33.345>And <00:15:33.472>I <00:15:33.599>guess, <00:15:33.726>okay, <00:15:33.854>maybe <00:15:33.981>we <00:15:34.108>say <00:15:34.235>agent <00:15:34.363>two <00:15:34.490>might <00:15:34.617>be <00:15:34.745>better

361
00:15:36.525 --> 00:15:38.885
<00:15:36.525>because <00:15:36.758>for <00:15:36.991>us <00:15:37.225>it's <00:15:37.458>more <00:15:37.691>important <00:15:37.925>that

362
00:15:39.395 --> 00:15:42.565
<00:15:39.395>more <00:15:39.550>users <00:15:39.706>are <00:15:39.861>successful <00:15:40.017>than <00:15:40.172>one <00:15:40.328>user <00:15:40.483>is <00:15:40.639>very <00:15:40.795>successful.

363
00:15:43.305 --> 00:15:44.725
<00:15:43.305>Of <00:15:43.487>course, <00:15:43.670>it's <00:15:43.852>not <00:15:44.035>that <00:15:44.217>easy <00:15:44.400>to <00:15:44.583>say

364
00:15:45.195 --> 00:15:46.405
<00:15:45.195>what <00:15:45.467>are <00:15:45.739>the <00:15:46.011>cost <00:15:46.284>differences?

365
00:15:46.745 --> 00:15:49.085
<00:15:46.745>How <00:15:46.978>are <00:15:47.211>they <00:15:47.445>different <00:15:47.678>to <00:15:47.911>each <00:15:48.145>other?

366
00:15:49.185 --> 00:15:52.605
<00:15:49.185>But <00:15:49.359>with <00:15:49.535>agent <00:15:49.709>evaluation, <00:15:49.885>we <00:15:50.060>are <00:15:50.234>really <00:15:50.410>changing <00:15:50.584>from,

367
00:15:52.675 --> 00:15:56.885
<00:15:52.675>okay, <00:15:52.802>what <00:15:52.929>works <00:15:53.056>on <00:15:53.184>one <00:15:53.311>attempt <00:15:53.438>to <00:15:53.565>how <00:15:53.693>can <00:15:53.820>we <00:15:53.947>make <00:15:54.074>sure

368
00:15:56.885 --> 00:15:58.245
<00:15:56.885>that <00:15:57.089>it <00:15:57.293>works <00:15:57.497>all <00:15:57.701>of <00:15:57.905>the <00:15:58.109>time?

369
00:15:59.705 --> 00:16:01.325
<00:15:59.705>So <00:16:00.171>context <00:16:00.638>engineering <00:16:01.105>is

370
00:16:02.515 --> 00:16:05.285
<00:16:02.515>something <00:16:02.795>I <00:16:03.074>included <00:16:03.355>two <00:16:03.635>days <00:16:03.915>ago.

371
00:16:05.435 --> 00:16:06.925
<00:16:05.435>It's <00:16:05.703>like <00:16:05.971>a <00:16:06.239>very <00:16:06.507>new <00:16:06.776>term

372
00:16:06.985 --> 00:16:08.805
<00:16:06.985>that's <00:16:07.185>coming <00:16:07.385>up <00:16:07.585>in <00:16:07.785>the <00:16:07.985>last <00:16:08.185>two <00:16:08.385>weeks.

373
00:16:09.385 --> 00:16:13.045
<00:16:09.385>Um, <00:16:09.560>it <00:16:09.735>is <00:16:09.910>something <00:16:10.085>that <00:16:10.260>it <00:16:10.435>extends <00:16:10.610>prompt <00:16:10.785>engineering

374
00:16:13.045 --> 00:16:16.045
<00:16:13.045>and <00:16:13.161>I <00:16:13.278>fit, <00:16:13.395>it's, <00:16:13.511>it <00:16:13.628>fits <00:16:13.745>very <00:16:13.861>well <00:16:13.978>in <00:16:14.094>how <00:16:14.211>we <00:16:14.328>are <00:16:14.444>going

375
00:16:16.045 --> 00:16:19.405
<00:16:16.045>to <00:16:16.245>build <00:16:16.444>agent <00:16:16.645>because <00:16:16.844>prompt <00:16:17.045>engineering <00:16:17.245>was <00:16:17.444>most

376
00:16:19.405 --> 00:16:22.365
<00:16:19.405>of <00:16:19.521>the <00:16:19.638>time <00:16:19.755>the <00:16:19.871>task <00:16:19.988>where <00:16:20.105>we <00:16:20.221>try <00:16:20.338>to <00:16:20.454>find <00:16:20.571>a <00:16:20.688>very <00:16:20.805>good,

377
00:16:22.505 --> 00:16:26.245
<00:16:22.505>simple <00:16:22.680>template <00:16:22.855>we <00:16:23.030>can <00:16:23.205>use <00:16:23.380>for <00:16:23.555>our <00:16:23.730>use <00:16:23.905>case

378
00:16:26.465 --> 00:16:29.325
<00:16:26.465>and <00:16:26.665>then <00:16:26.865>provide <00:16:27.065>different <00:16:27.265>variables <00:16:27.465>when <00:16:27.665>running <00:16:27.865>our

379
00:16:29.505 --> 00:16:30.645
<00:16:29.505>um, <00:16:30.018>LLM <00:16:30.531>call.

380
00:16:31.185 --> 00:16:33.005
<00:16:31.185>But <00:16:31.464>that <00:16:31.744>doesn't <00:16:32.025>longer <00:16:32.305>work, <00:16:32.584>right?

381
00:16:33.005 --> 00:16:36.365
<00:16:33.005>With <00:16:33.132>agents, <00:16:33.259>we <00:16:33.386>have <00:16:33.514>tools, <00:16:33.641>we <00:16:33.768>have <00:16:33.895>memories, <00:16:34.023>we <00:16:34.150>have <00:16:34.277>a <00:16:34.405>lot

382
00:16:36.365 --> 00:16:38.045
<00:16:36.365>of <00:16:36.598>other <00:16:36.831>things <00:16:37.065>that <00:16:37.298>we <00:16:37.531>now <00:16:37.765>need

383
00:16:38.045 --> 00:16:40.525
<00:16:38.045>to <00:16:38.245>consider <00:16:38.444>when <00:16:38.645>like <00:16:38.844>running <00:16:39.045>our <00:16:39.245>LLM <00:16:39.444>call

384
00:16:40.665 --> 00:16:42.685
<00:16:40.665>and <00:16:41.015>contact <00:16:41.365>contact, <00:16:41.714>uh, <00:16:42.064>sorry,

385
00:16:42.715 --> 00:16:45.125
<00:16:42.715>context <00:16:42.915>engineering <00:16:43.115>is <00:16:43.315>a <00:16:43.515>term <00:16:43.715>that <00:16:43.915>comes <00:16:44.115>up

386
00:16:45.145 --> 00:16:48.805
<00:16:45.145>and <00:16:45.425>is <00:16:45.704>defined <00:16:45.985>as <00:16:46.265>a <00:16:46.545>way

387
00:16:48.945 --> 00:16:51.805
<00:16:48.945>or <00:16:49.120>building <00:16:49.295>a <00:16:49.470>system <00:16:49.645>that <00:16:49.820>provides <00:16:49.995>the <00:16:50.170>right <00:16:50.345>information.

388
00:16:51.945 --> 00:16:54.285
<00:16:51.945>So <00:16:52.225>our <00:16:52.505>prompt, <00:16:52.785>the <00:16:53.065>right <00:16:53.345>tools

389
00:16:54.865 --> 00:16:56.525
<00:16:54.865>in <00:16:55.065>the <00:16:55.265>right <00:16:55.465>format <00:16:55.665>at <00:16:55.865>the <00:16:56.065>right <00:16:56.265>time

390
00:16:56.555 --> 00:16:59.085
<00:16:56.555>because <00:16:57.255>sometimes <00:16:57.954>information,

391
00:16:59.085 --> 00:17:02.525
<00:16:59.085>which <00:16:59.240>I <00:16:59.396>might <00:16:59.551>include <00:16:59.707>into <00:16:59.862>my <00:17:00.018>standard <00:17:00.173>prompt <00:17:00.329>template <00:17:00.485>are

392
00:17:02.525 --> 00:17:05.845
<00:17:02.525>not <00:17:02.699>helping <00:17:02.875>for, <00:17:03.050>um, <00:17:03.225>the <00:17:03.400>current <00:17:03.574>user <00:17:03.750>of <00:17:03.925>Prompt.

393
00:17:06.145 --> 00:17:09.045
<00:17:06.145>If <00:17:06.319>we <00:17:06.494>look <00:17:06.670>at <00:17:06.845>an <00:17:07.020>example <00:17:07.194>when, <00:17:07.369>for <00:17:07.545>example,

394
00:17:09.045 --> 00:17:10.565
<00:17:09.045>we <00:17:09.318>are <00:17:09.592>building <00:17:09.865>an <00:17:10.139>email <00:17:10.413>agent

395
00:17:11.265 --> 00:17:14.445
<00:17:11.265>and <00:17:11.420>you <00:17:11.576>receive <00:17:11.731>an <00:17:11.887>email <00:17:12.042>with <00:17:12.198>asking, <00:17:12.353>Hey <00:17:12.509>Phillip, <00:17:12.665>um,

396
00:17:14.585 --> 00:17:15.605
<00:17:14.585>are <00:17:14.814>you <00:17:15.044>free <00:17:15.273>on <00:17:15.502>Tuesday?

397
00:17:15.925 --> 00:17:18.485
<00:17:15.925>I <00:17:16.065>would <00:17:16.204>like <00:17:16.345>to <00:17:16.484>book <00:17:16.625>a <00:17:16.764>meeting <00:17:16.905>on <00:17:17.044>in <00:17:17.185>this <00:17:17.325>case,

398
00:17:18.665 --> 00:17:20.005
<00:17:18.665>it <00:17:19.067>is <00:17:19.469>very <00:17:19.871>important

399
00:17:20.115 --> 00:17:23.645
<00:17:20.115>what <00:17:20.290>information <00:17:20.464>the <00:17:20.640>agent <00:17:20.815>has <00:17:20.990>available <00:17:21.165>to <00:17:21.339>respond, <00:17:21.515>right?

400
00:17:23.865 --> 00:17:25.685
<00:17:23.865>If <00:17:24.214>it <00:17:24.565>only <00:17:24.915>can <00:17:25.265>call

401
00:17:25.705 --> 00:17:29.405
<00:17:25.704>or <00:17:25.845>send <00:17:25.984>an <00:17:26.125>email, <00:17:26.264>most <00:17:26.405>likely <00:17:26.544>it <00:17:26.685>would <00:17:26.824>respond <00:17:26.964>with <00:17:27.105>like,

402
00:17:29.835 --> 00:17:31.325
<00:17:29.835>okay, <00:17:30.058>can <00:17:30.282>you <00:17:30.505>tell <00:17:30.729>me <00:17:30.952>more <00:17:31.176>about

403
00:17:31.355 --> 00:17:32.565
<00:17:31.355>what <00:17:31.572>would <00:17:31.790>you <00:17:32.008>like <00:17:32.226>to <00:17:32.444>meet?

404
00:17:32.945 --> 00:17:35.685
<00:17:32.945>So <00:17:33.100>it's <00:17:33.256>like <00:17:33.411>going <00:17:33.567>back <00:17:33.722>and <00:17:33.878>forth <00:17:34.033>with <00:17:34.189>the <00:17:34.345>email.

405
00:17:35.905 --> 00:17:39.205
<00:17:35.905>But <00:17:36.060>if <00:17:36.216>my <00:17:36.371>agent <00:17:36.527>has <00:17:36.682>access <00:17:36.838>to <00:17:36.993>my <00:17:37.149>calendar <00:17:37.305>information,

406
00:17:39.655 --> 00:17:43.285
<00:17:39.655>knows <00:17:39.810>my <00:17:39.966>preferences, <00:17:40.121>when <00:17:40.277>I <00:17:40.432>want <00:17:40.588>to <00:17:40.743>have <00:17:40.899>meetings, <00:17:41.055>knows,

407
00:17:43.665 --> 00:17:46.725
<00:17:43.665>um, <00:17:43.805>that <00:17:43.945>I <00:17:44.085>so <00:17:44.225>like <00:17:44.365>to <00:17:44.504>have <00:17:44.645>a <00:17:44.784>document <00:17:44.925>in <00:17:45.065>advance,

408
00:17:47.065 --> 00:17:50.805
<00:17:47.065>it <00:17:47.220>can <00:17:47.376>generate <00:17:47.531>a <00:17:47.687>much <00:17:47.842>better <00:17:47.998>respond <00:17:48.153>to <00:17:48.309>the <00:17:48.465>request

409
00:17:50.805 --> 00:17:52.925
<00:17:50.805>of <00:17:50.980>the <00:17:51.155>user <00:17:51.330>saying, <00:17:51.505>Hey, <00:17:51.680>due <00:17:51.855>stay <00:17:52.030>is <00:17:52.205>bad

410
00:17:52.925 --> 00:17:54.165
<00:17:52.925>because <00:17:53.148>Philip <00:17:53.371>is <00:17:53.594>still <00:17:53.817>in <00:17:54.041>Berlin.

411
00:17:54.345 --> 00:17:55.565
<00:17:54.345>Can <00:17:54.564>you <00:17:54.784>share <00:17:55.003>a <00:17:55.223>document <00:17:55.443>what,

412
00:17:55.565 --> 00:17:56.765
<00:17:55.565>which <00:17:55.745>you <00:17:55.925>would <00:17:56.105>like <00:17:56.285>to <00:17:56.465>meet <00:17:56.645>about?

413
00:17:57.145 --> 00:18:01.445
<00:17:57.145>So <00:17:57.378>engineering <00:17:57.611>our <00:17:57.845>context, <00:17:58.078>which <00:17:58.311>provides <00:17:58.545>all

414
00:18:01.445 --> 00:18:05.445
<00:18:01.445>of <00:18:01.620>the <00:18:01.794>information <00:18:01.970>to <00:18:02.145>our <00:18:02.320>LLM <00:18:02.495>to <00:18:02.669>correctly <00:18:02.845>reason

415
00:18:06.105 --> 00:18:09.045
<00:18:06.105>is <00:18:06.280>the <00:18:06.455>skill <00:18:06.630>we <00:18:06.805>are <00:18:06.980>probably <00:18:07.155>going <00:18:07.330>to <00:18:07.505>optimize

416
00:18:09.045 --> 00:18:11.965
<00:18:09.045>and <00:18:09.200>focus <00:18:09.356>on <00:18:09.511>for <00:18:09.667>the <00:18:09.822>next <00:18:09.978>few <00:18:10.133>months <00:18:10.289>at <00:18:10.445>least.

417
00:18:11.985 --> 00:18:14.285
<00:18:11.985>And <00:18:12.140>I <00:18:12.296>try <00:18:12.451>to <00:18:12.607>visualize <00:18:12.762>it <00:18:12.918>a <00:18:13.073>bit <00:18:13.229>more <00:18:13.385>on

418
00:18:14.395 --> 00:18:16.565
<00:18:14.395>what <00:18:14.628>is <00:18:14.861>all <00:18:15.095>included <00:18:15.328>in <00:18:15.561>our <00:18:15.795>context.

419
00:18:16.745 --> 00:18:17.845
<00:18:16.745>So <00:18:16.992>we <00:18:17.240>have <00:18:17.487>the <00:18:17.735>instruction

420
00:18:17.845 --> 00:18:19.485
<00:18:17.845>and <00:18:18.195>system <00:18:18.545>prompts, <00:18:18.895>which <00:18:19.245>defines

421
00:18:19.485 --> 00:18:21.205
<00:18:19.485>what <00:18:19.718>our <00:18:19.951>LLM <00:18:20.185>or <00:18:20.418>agent <00:18:20.651>should <00:18:20.885>do.

422
00:18:21.425 --> 00:18:24.045
<00:18:21.425>We <00:18:21.600>have <00:18:21.774>the <00:18:21.950>long <00:18:22.125>term <00:18:22.300>memory <00:18:22.475>with, <00:18:22.649>um, <00:18:22.825>history

423
00:18:24.045 --> 00:18:25.165
<00:18:24.045>and <00:18:24.549>user <00:18:25.053>preferences.

424
00:18:25.185 --> 00:18:28.405
<00:18:25.185>We <00:18:25.340>have <00:18:25.496>retrieved <00:18:25.651>information <00:18:25.807>maybe <00:18:25.962>from, <00:18:26.118>uh, <00:18:26.273>real <00:18:26.429>time, <00:18:26.585>uh,

425
00:18:28.725 --> 00:18:29.765
<00:18:28.725>endpoints <00:18:29.037>or <00:18:29.349>Google <00:18:29.661>search.

426
00:18:30.065 --> 00:18:32.525
<00:18:30.065>We <00:18:30.192>have <00:18:30.319>the <00:18:30.446>user <00:18:30.574>prompt, <00:18:30.701>we <00:18:30.828>have <00:18:30.955>all <00:18:31.083>of <00:18:31.210>the <00:18:31.337>available <00:18:31.465>tools.

427
00:18:32.625 --> 00:18:34.565
<00:18:32.625>We <00:18:32.825>can <00:18:33.025>define <00:18:33.225>how <00:18:33.425>the <00:18:33.625>output <00:18:33.825>should <00:18:34.025>look.

428
00:18:34.745 --> 00:18:37.845
<00:18:34.745>So <00:18:34.900>prompt <00:18:35.056>engineering <00:18:35.211>is <00:18:35.367>only <00:18:35.522>one <00:18:35.678>small <00:18:35.833>part <00:18:35.989>of <00:18:36.145>it

429
00:18:37.985 --> 00:18:41.205
<00:18:37.985>and <00:18:38.159>we <00:18:38.334>are <00:18:38.510>now <00:18:38.685>going <00:18:38.860>beyond <00:18:39.034>with <00:18:39.209>um, <00:18:39.385>agents.

430
00:18:41.635 --> 00:18:45.445
<00:18:41.635>Cool. <00:18:41.790>So <00:18:41.946>let's <00:18:42.101>look <00:18:42.257>a <00:18:42.412>bit <00:18:42.568>at <00:18:42.723>at <00:18:42.879>practical <00:18:43.035>examples.

431
00:18:45.865 --> 00:18:48.685
<00:18:45.865>So <00:18:46.040>for <00:18:46.215>do, <00:18:46.390>uh, <00:18:46.565>for <00:18:46.740>the <00:18:46.915>one <00:18:47.090>of <00:18:47.265>you

432
00:18:48.685 --> 00:18:50.605
<00:18:48.685>who <00:18:49.034>never <00:18:49.385>use <00:18:49.735>Google <00:18:50.085>Gemini

433
00:18:50.905 --> 00:18:53.605
<00:18:50.905>or <00:18:51.045>never <00:18:51.185>tested <00:18:51.325>it, <00:18:51.465>the <00:18:51.605>easiest <00:18:51.745>way <00:18:51.885>is <00:18:52.024>if <00:18:52.165>you <00:18:52.305>go

434
00:18:53.605 --> 00:18:57.405
<00:18:53.605>to <00:18:53.760>AI <00:18:53.916>studio <00:18:54.071>or <00:18:54.227>AI <00:18:54.382>studio.google.com, <00:18:54.538>you <00:18:54.693>can <00:18:54.849>log <00:18:55.005>in

435
00:18:57.405 --> 00:19:00.205
<00:18:57.405>with <00:18:57.580>your <00:18:57.754>Google <00:18:57.930>account, <00:18:58.105>um, <00:18:58.280>no <00:18:58.455>credit <00:18:58.629>card <00:18:58.805>required.

436
00:19:00.385 --> 00:19:02.085
<00:19:00.385>You <00:19:00.665>can <00:19:00.945>generate <00:19:01.225>an <00:19:01.504>API <00:19:01.785>key.

437
00:19:02.145 --> 00:19:04.245
<00:19:02.145>We <00:19:02.378>have <00:19:02.611>a <00:19:02.845>very <00:19:03.078>generous <00:19:03.311>free <00:19:03.545>tier.

438
00:19:04.345 --> 00:19:06.445
<00:19:04.345>So <00:19:04.520>you <00:19:04.695>can <00:19:04.870>test <00:19:05.045>all <00:19:05.220>of <00:19:05.395>the <00:19:05.570>Gemini <00:19:05.745>models,

439
00:19:06.595 --> 00:19:09.045
<00:19:06.595>even <00:19:06.875>Gemini <00:19:07.155>2.5 <00:19:07.435>Pro <00:19:07.715>for <00:19:07.995>free

440
00:19:09.385 --> 00:19:11.165
<00:19:09.385>and <00:19:09.735>you <00:19:10.085>can <00:19:10.435>start <00:19:10.785>experimenting.

441
00:19:11.385 --> 00:19:14.165
<00:19:11.385>The <00:19:11.525>easiest <00:19:11.665>way <00:19:11.805>to <00:19:11.945>experiment <00:19:12.085>is <00:19:12.225>of <00:19:12.365>course <00:19:12.504>in <00:19:12.645>AI <00:19:12.785>Studio,

442
00:19:14.595 --> 00:19:17.045
<00:19:14.595>it's <00:19:14.735>a <00:19:14.875>very <00:19:15.015>similar <00:19:15.155>to <00:19:15.295>chat <00:19:15.435>GBT, <00:19:15.575>it's <00:19:15.715>an <00:19:15.855>i <00:19:15.995>ui.

443
00:19:17.305 --> 00:19:19.565
<00:19:17.305>You <00:19:17.480>have <00:19:17.655>access <00:19:17.830>to <00:19:18.005>the <00:19:18.180>models <00:19:18.355>you <00:19:18.530>have, <00:19:18.705>um,

444
00:19:19.765 --> 00:19:22.605
<00:19:19.765>I <00:19:19.940>would <00:19:20.115>say <00:19:20.290>much <00:19:20.465>more <00:19:20.640>developer, <00:19:20.815>um, <00:19:20.990>looking <00:19:21.165>interface.

445
00:19:22.625 --> 00:19:24.805
<00:19:22.625>You <00:19:22.825>can <00:19:23.025>control <00:19:23.225>the <00:19:23.425>temperature <00:19:23.625>in <00:19:23.825>other <00:19:24.025>parameters

446
00:19:24.985 --> 00:19:26.845
<00:19:24.985>and <00:19:25.218>you <00:19:25.451>can <00:19:25.685>directly <00:19:25.918>start <00:19:26.151>prompting <00:19:26.385>it.

447
00:19:27.025 --> 00:19:29.165
<00:19:27.025>You <00:19:27.305>can, <00:19:27.585>um, <00:19:27.865>use <00:19:28.145>tools <00:19:28.425>directly.

448
00:19:29.165 --> 00:19:31.245
<00:19:29.165>So <00:19:29.365>I <00:19:29.565>asked <00:19:29.764>to <00:19:29.965>one <00:19:30.165>the <00:19:30.365>Euro <00:19:30.565>2024.

449
00:19:31.885 --> 00:19:34.165
<00:19:31.885>Normally <00:19:32.085>the <00:19:32.285>model, <00:19:32.485>um, <00:19:32.685>says <00:19:32.885>it <00:19:33.085>cannot <00:19:33.285>respond

450
00:19:34.165 --> 00:19:35.805
<00:19:34.165>because <00:19:34.445>the <00:19:34.725>Euro <00:19:35.004>2024 <00:19:35.284>didn't <00:19:35.565>happen.

451
00:19:36.225 --> 00:19:39.365
<00:19:36.225>But <00:19:36.380>I <00:19:36.536>hooked <00:19:36.691>it <00:19:36.847>into <00:19:37.002>the, <00:19:37.158>uh, <00:19:37.313>Google <00:19:37.469>search <00:19:37.625>tool

452
00:19:39.545 --> 00:19:42.485
<00:19:39.545>and <00:19:39.700>it <00:19:39.856>used <00:19:40.011>Google <00:19:40.167>search, <00:19:40.322>uh, <00:19:40.478>to <00:19:40.633>generate <00:19:40.789>my <00:19:40.945>response.

453
00:19:42.605 --> 00:19:45.125
<00:19:42.605>I <00:19:42.721>can <00:19:42.838>also <00:19:42.955>look <00:19:43.071>at <00:19:43.188>the, <00:19:43.305>the <00:19:43.421>thoughts <00:19:43.538>of <00:19:43.655>the <00:19:43.771>model <00:19:43.888>to <00:19:44.005>see,

454
00:19:45.155 --> 00:19:46.485
<00:19:45.155>okay, <00:19:45.454>what <00:19:45.753>has <00:19:46.052>it <00:19:46.351>done?

455
00:19:47.465 --> 00:19:50.885
<00:19:47.465>But <00:19:47.581>if <00:19:47.698>you <00:19:47.814>don't <00:19:47.931>want <00:19:48.048>to <00:19:48.165>only <00:19:48.281>experiment <00:19:48.398>in <00:19:48.514>the <00:19:48.631>ui, <00:19:48.748>you <00:19:48.865>can

456
00:19:50.885 --> 00:19:53.725
<00:19:50.885>of <00:19:51.060>course <00:19:51.235>use <00:19:51.410>the <00:19:51.585>Google, <00:19:51.760>uh, <00:19:51.935>gene <00:19:52.110>I <00:19:52.285>SDKs,

457
00:19:53.825 --> 00:19:57.165
<00:19:53.825>that's <00:19:54.000>our <00:19:54.175>first <00:19:54.350>party <00:19:54.525>SDKs <00:19:54.700>to <00:19:54.875>work <00:19:55.050>with, <00:19:55.225>uh,

458
00:19:57.165 --> 00:20:00.125
<00:19:57.165>the <00:19:57.365>Gemini <00:19:57.565>API, <00:19:57.764>uh, <00:19:57.965>that's <00:19:58.165>a <00:19:58.365>code <00:19:58.565>snippet.

459
00:20:00.225 --> 00:20:03.245
<00:20:00.225>Um, <00:20:00.380>basically <00:20:00.536>for <00:20:00.691>all <00:20:00.847>of <00:20:01.002>the <00:20:01.158>code <00:20:01.313>you <00:20:01.469>need <00:20:01.625>to

460
00:20:04.065 --> 00:20:05.965
<00:20:04.065>create <00:20:04.345>use <00:20:04.625>to <00:20:04.905>build <00:20:05.185>an <00:20:05.465>agent.

461
00:20:06.225 --> 00:20:08.285
<00:20:06.225>Um, <00:20:06.458>so <00:20:06.691>the <00:20:06.925>example <00:20:07.158>is <00:20:07.391>very <00:20:07.625>simple.

462
00:20:08.625 --> 00:20:11.645
<00:20:08.625>We <00:20:08.800>have <00:20:08.975>a <00:20:09.150>Python <00:20:09.325>function <00:20:09.500>which <00:20:09.675>defines <00:20:09.850>our <00:20:10.025>tool.

463
00:20:11.745 --> 00:20:13.605
<00:20:11.745>In <00:20:11.978>this <00:20:12.211>case <00:20:12.445>get <00:20:12.678>current <00:20:12.911>weather <00:20:13.145>information.

464
00:20:13.865 --> 00:20:17.765
<00:20:13.865>We <00:20:14.040>have <00:20:14.215>a <00:20:14.390>client, <00:20:14.565>um, <00:20:14.740>we <00:20:14.915>add <00:20:15.090>our <00:20:15.265>tools

465
00:20:17.865 --> 00:20:19.645
<00:20:17.865>to <00:20:18.331>our <00:20:18.798>generation <00:20:19.265>config,

466
00:20:19.645 --> 00:20:21.005
<00:20:19.645>and <00:20:19.819>then <00:20:19.994>I <00:20:20.169>call <00:20:20.344>can <00:20:20.519>call <00:20:20.694>the <00:20:20.869>model

467
00:20:21.035 --> 00:20:22.525
<00:20:21.035>with <00:20:21.258>like <00:20:21.482>what's <00:20:21.705>the <00:20:21.929>temperature <00:20:22.152>in <00:20:22.376>Boston?

468
00:20:22.905 --> 00:20:25.245
<00:20:22.905>And <00:20:23.105>the <00:20:23.305>SDK <00:20:23.504>handles <00:20:23.705>all <00:20:23.905>of <00:20:24.105>the, <00:20:24.305>all

469
00:20:25.245 --> 00:20:26.725
<00:20:25.245>of <00:20:25.511>the <00:20:25.777>obstruction <00:20:26.044>for <00:20:26.310>tool <00:20:26.577>calling.

470
00:20:26.865 --> 00:20:29.085
<00:20:26.865>So <00:20:27.040>if <00:20:27.215>the <00:20:27.390>model <00:20:27.565>decides <00:20:27.740>to <00:20:27.915>use <00:20:28.090>a <00:20:28.265>tool,

471
00:20:29.665 --> 00:20:33.405
<00:20:29.665>it <00:20:29.805>calls <00:20:29.945>our <00:20:30.085>Python <00:20:30.225>function, <00:20:30.365>uses <00:20:30.504>the <00:20:30.645>return, <00:20:30.784>sends <00:20:30.925>it <00:20:31.065>back

472
00:20:33.405 --> 00:20:35.925
<00:20:33.405>to <00:20:33.580>the <00:20:33.754>model, <00:20:33.930>the <00:20:34.105>model <00:20:34.280>generates <00:20:34.455>a <00:20:34.629>much <00:20:34.805>nicer,

473
00:20:36.105 --> 00:20:37.165
<00:20:36.105>uh, <00:20:36.581>user <00:20:37.059>response.

474
00:20:37.385 --> 00:20:40.085
<00:20:37.385>And <00:20:37.512>we <00:20:37.639>get <00:20:37.766>back <00:20:37.894>all <00:20:38.021>of <00:20:38.148>that <00:20:38.275>in <00:20:38.403>our, <00:20:38.530>uh, <00:20:38.657>response <00:20:38.785>text.

475
00:20:40.585 --> 00:20:41.685
<00:20:40.585>But <00:20:40.832>uh, <00:20:41.080>that's <00:20:41.327>not <00:20:41.575>all.

476
00:20:41.945 --> 00:20:43.485
<00:20:41.945>We <00:20:42.222>are <00:20:42.499>working <00:20:42.776>actively <00:20:43.053>with <00:20:43.331>the,

477
00:20:43.485 --> 00:20:46.045
<00:20:43.485>with <00:20:43.640>the <00:20:43.796>whole <00:20:43.951>open <00:20:44.107>source <00:20:44.262>ecosystem <00:20:44.418>to <00:20:44.573>make <00:20:44.729>sure <00:20:44.885>that

478
00:20:46.685 --> 00:20:48.645
<00:20:46.685>everyone <00:20:47.034>can <00:20:47.385>build <00:20:47.735>agents <00:20:48.085>workflows

479
00:20:48.645 --> 00:20:51.085
<00:20:48.645>and <00:20:48.800>uh, <00:20:48.956>AI <00:20:49.111>applications <00:20:49.267>with <00:20:49.422>the <00:20:49.578>tools <00:20:49.733>of <00:20:49.889>their <00:20:50.045>joy.

480
00:20:51.105 --> 00:20:52.445
<00:20:51.105>So <00:20:51.306>we <00:20:51.507>are <00:20:51.708>working <00:20:51.908>with <00:20:52.110>Link <00:20:52.311>chain

481
00:20:52.445 --> 00:20:53.925
<00:20:52.445>and <00:20:52.635>Land <00:20:52.825>Craft <00:20:53.015>to <00:20:53.206>make <00:20:53.396>it <00:20:53.586>super <00:20:53.777>easy

482
00:20:53.985 --> 00:20:55.125
<00:20:53.985>to <00:20:54.241>use <00:20:54.498>the <00:20:54.754>Gemini <00:20:55.011>models.

483
00:20:55.625 --> 00:20:58.925
<00:20:55.625>We <00:20:55.752>are <00:20:55.879>working <00:20:56.006>with <00:20:56.134>Q <00:20:56.261>Crew <00:20:56.388>AI <00:20:56.515>to <00:20:56.643>make <00:20:56.770>it <00:20:56.897>super <00:20:57.025>easy

484
00:20:58.985 --> 00:21:00.405
<00:20:58.985>to <00:20:59.304>build <00:20:59.624>a <00:20:59.943>multi-agent <00:21:00.263>system.

485
00:21:00.465 --> 00:21:01.845
<00:21:00.465>So <00:21:00.672>if <00:21:00.879>you <00:21:01.086>are <00:21:01.293>now <00:21:01.500>interested, <00:21:01.707>okay,

486
00:21:01.845 --> 00:21:03.485
<00:21:01.845>how <00:21:02.078>can <00:21:02.311>I <00:21:02.545>build <00:21:02.778>a <00:21:03.011>multi-agent <00:21:03.245>system?

487
00:21:03.995 --> 00:21:06.045
<00:21:03.995>Crew <00:21:04.195>is <00:21:04.395>a <00:21:04.594>a <00:21:04.794>very <00:21:04.995>cool, <00:21:05.195>um, <00:21:05.395>library

488
00:21:06.185 --> 00:21:07.485
<00:21:06.185>and <00:21:06.379>definitely <00:21:06.574>you <00:21:06.770>should <00:21:06.965>check <00:21:07.159>it <00:21:07.355>out.

489
00:21:07.745 --> 00:21:09.885
<00:21:07.745>We <00:21:07.945>are <00:21:08.145>working <00:21:08.344>with <00:21:08.544>LAMA <00:21:08.745>Index <00:21:08.945>to, <00:21:09.145>uh,

490
00:21:09.885 --> 00:21:13.125
<00:21:09.885>help <00:21:10.085>you <00:21:10.285>build <00:21:10.485>much <00:21:10.685>easier <00:21:10.885>document <00:21:11.085>processing, <00:21:11.285>uh,

491
00:21:13.445 --> 00:21:18.365
<00:21:13.445>workflows <00:21:13.620>and <00:21:13.794>many, <00:21:13.970>many <00:21:14.145>more <00:21:14.320>libraries, <00:21:14.495>uh, <00:21:14.669>companies, <00:21:14.845>um,

492
00:21:18.585 --> 00:21:20.245
<00:21:18.585>to <00:21:18.818>help <00:21:19.051>you <00:21:19.285>make <00:21:19.518>it <00:21:19.751>super <00:21:19.985>easy

493
00:21:20.345 --> 00:21:23.285
<00:21:20.345>and <00:21:20.625>simple <00:21:20.905>to <00:21:21.185>build, <00:21:21.465>uh, <00:21:21.745>agents

494
00:21:23.385 --> 00:21:25.565
<00:21:23.385>and <00:21:23.735>AI <00:21:24.085>applications <00:21:24.435>with <00:21:24.785>Gemini.

495
00:21:26.585 --> 00:21:29.405
<00:21:26.585>Um, <00:21:26.760>if <00:21:26.935>you <00:21:27.110>are <00:21:27.285>looking <00:21:27.460>for <00:21:27.635>concrete <00:21:27.810>examples, <00:21:27.985>uh,

496
00:21:29.405 --> 00:21:31.845
<00:21:29.405>we <00:21:29.545>have <00:21:29.685>two <00:21:29.825>very <00:21:29.965>cool <00:21:30.105>ones <00:21:30.245>which <00:21:30.385>are <00:21:30.524>both <00:21:30.665>open <00:21:30.805>source

497
00:21:31.905 --> 00:21:33.925
<00:21:31.905>and <00:21:32.185>available <00:21:32.465>on <00:21:32.745>our, <00:21:33.024>uh, <00:21:33.305>GitHub

498
00:21:34.025 --> 00:21:36.805
<00:21:34.025>or <00:21:34.305>um, <00:21:34.585>organization <00:21:34.865>under <00:21:35.145>Google <00:21:35.425>Gemini.

499
00:21:37.065 --> 00:21:40.565
<00:21:37.065>That's, <00:21:37.265>uh, <00:21:37.465>an <00:21:37.665>open <00:21:37.865>source <00:21:38.065>deep <00:21:38.265>research <00:21:38.465>agent.

500
00:21:40.585 --> 00:21:42.885
<00:21:40.585>We <00:21:40.818>build <00:21:41.051>using <00:21:41.285>land <00:21:41.518>graph <00:21:41.751>and <00:21:41.985>Gemini.

501
00:21:43.065 --> 00:21:45.365
<00:21:43.065>It <00:21:43.265>uses <00:21:43.465>the <00:21:43.665>native <00:21:43.865>tool <00:21:44.065>for <00:21:44.265>Google <00:21:44.465>search

502
00:21:45.745 --> 00:21:47.205
<00:21:45.745>and <00:21:46.073>basically <00:21:46.401>does <00:21:46.730>the <00:21:47.059>same

503
00:21:47.225 --> 00:21:50.165
<00:21:47.225>as <00:21:47.399>the <00:21:47.574>Google <00:21:47.750>Deep <00:21:47.925>research <00:21:48.100>in <00:21:48.274>the <00:21:48.449>Gemini <00:21:48.625>app.

504
00:21:50.395 --> 00:21:53.365
<00:21:50.395>Very <00:21:50.535>cool. <00:21:50.675>You <00:21:50.815>can <00:21:50.955>download <00:21:51.095>it, <00:21:51.235>just <00:21:51.375>need <00:21:51.514>Gemini <00:21:51.655>API <00:21:51.795>key.

505
00:21:53.365 --> 00:21:54.885
<00:21:53.365>It <00:21:53.593>works <00:21:53.821>with <00:21:54.049>the <00:21:54.277>free <00:21:54.505>tier <00:21:54.733>and

506
00:21:54.885 --> 00:21:56.005
<00:21:54.885>you <00:21:55.137>can <00:21:55.389>start <00:21:55.641>playing <00:21:55.893>around.

507
00:21:56.505 --> 00:21:58.685
<00:21:56.505>And <00:21:56.738>then <00:21:56.971>of <00:21:57.205>course, <00:21:57.438>uh, <00:21:57.671>Gemini <00:21:57.905>CLI,

508
00:21:58.685 --> 00:22:00.325
<00:21:58.685>which <00:21:58.885>we <00:21:59.085>launched <00:21:59.284>a <00:21:59.485>couple <00:21:59.685>of <00:21:59.885>weeks <00:22:00.085>ago,

509
00:22:00.325 --> 00:22:03.725
<00:22:00.325>which <00:22:00.480>is <00:22:00.636>our <00:22:00.791>coding <00:22:00.947>agent <00:22:01.102>is <00:22:01.258>also <00:22:01.413>open <00:22:01.569>source, <00:22:01.725>um,

510
00:22:03.725 --> 00:22:06.685
<00:22:03.725>available <00:22:03.958>on <00:22:04.191>the <00:22:04.425>same <00:22:04.658>GitHub, <00:22:04.891>uh, <00:22:05.125>organization.

511
00:22:06.825 --> 00:22:09.765
<00:22:06.825>And <00:22:06.952>you <00:22:07.079>can <00:22:07.206>directly <00:22:07.334>look <00:22:07.461>into, <00:22:07.588>okay, <00:22:07.715>how <00:22:07.843>did <00:22:07.970>we <00:22:08.097>build, <00:22:08.225>um,

512
00:22:09.945 --> 00:22:12.285
<00:22:09.945>Gemini <00:22:10.178>CLI, <00:22:10.411>what <00:22:10.645>is <00:22:10.878>the <00:22:11.111>system <00:22:11.345>prompt,

513
00:22:12.295 --> 00:22:13.845
<00:22:12.295>which <00:22:12.527>are <00:22:12.760>the <00:22:12.992>tools <00:22:13.225>we <00:22:13.457>are <00:22:13.690>using

514
00:22:14.065 --> 00:22:15.845
<00:22:14.065>to <00:22:14.298>create <00:22:14.531>all <00:22:14.765>of <00:22:14.998>those <00:22:15.231>nice <00:22:15.465>experience?

515
00:22:16.065 --> 00:22:19.005
<00:22:16.065>And <00:22:16.192>of <00:22:16.319>course, <00:22:16.446>if <00:22:16.574>you <00:22:16.701>want <00:22:16.828>to <00:22:16.955>for <00:22:17.083>it <00:22:17.210>or <00:22:17.337>modify <00:22:17.465>it

516
00:22:19.005 --> 00:22:21.205
<00:22:19.005>or <00:22:19.205>use <00:22:19.405>it <00:22:19.605>internally, <00:22:19.805>that <00:22:20.005>that's <00:22:20.205>also <00:22:20.405>possible

517
00:22:21.585 --> 00:22:22.805
<00:22:21.585>and <00:22:21.859>very <00:22:22.134>cool <00:22:22.408>is <00:22:22.683>like

518
00:22:22.805 --> 00:22:25.885
<00:22:22.805>that <00:22:22.945>Gemini <00:22:23.085>CLI <00:22:23.225>is <00:22:23.365>the <00:22:23.505>first <00:22:23.645>agent <00:22:23.785>we <00:22:23.925>built <00:22:24.065>at <00:22:24.205>Google,

519
00:22:25.895 --> 00:22:28.525
<00:22:25.895>which <00:22:26.175>is <00:22:26.455>used <00:22:26.735>externally <00:22:27.014>and <00:22:27.295>internally.

520
00:22:29.065 --> 00:22:31.205
<00:22:29.065>So <00:22:29.345>the <00:22:29.625>Google <00:22:29.905>teams <00:22:30.185>from <00:22:30.465>YouTube

521
00:22:31.225 --> 00:22:34.885
<00:22:31.225>to <00:22:31.352>search <00:22:31.479>are <00:22:31.606>now <00:22:31.734>using <00:22:31.861>Gemini <00:22:31.988>CLI <00:22:32.115>as <00:22:32.243>part <00:22:32.370>of <00:22:32.497>their <00:22:32.625>workflow

522
00:22:34.885 --> 00:22:39.045
<00:22:34.885>to <00:22:35.118>improve <00:22:35.351>the <00:22:35.585>internal, <00:22:35.818>um, <00:22:36.051>code <00:22:36.285>infrastructure.

523
00:22:39.505 --> 00:22:42.245
<00:22:39.505>And <00:22:39.645>then <00:22:39.785>everyone <00:22:39.925>else <00:22:40.065>can <00:22:40.205>use <00:22:40.345>Gemini <00:22:40.485>the <00:22:40.625>same <00:22:40.765>way <00:22:40.905>basically

524
00:22:42.315 --> 00:22:44.725
<00:22:42.315>through <00:22:42.595>the <00:22:42.875>open <00:22:43.155>source, <00:22:43.435>uh, <00:22:43.715>repository.

525
00:22:45.875 --> 00:22:49.325
<00:22:45.875>Cool. <00:22:46.030>Before <00:22:46.186>we <00:22:46.341>go <00:22:46.497>to <00:22:46.652>the <00:22:46.808>questions, <00:22:46.963>um, <00:22:47.119>here <00:22:47.275>are

526
00:22:50.005 --> 00:22:51.325
<00:22:50.005>a <00:22:50.302>few <00:22:50.599>bullet <00:22:50.896>points <00:22:51.193>on

527
00:22:51.425 --> 00:22:54.125
<00:22:51.425>how <00:22:51.600>I <00:22:51.774>think <00:22:51.950>the <00:22:52.125>future <00:22:52.300>might <00:22:52.475>evolve <00:22:52.649>with <00:22:52.825>agents.

528
00:22:54.225 --> 00:22:55.725
<00:22:54.225>So <00:22:54.449>I <00:22:54.675>believe <00:22:54.899>that <00:22:55.125>we <00:22:55.350>will <00:22:55.574>have

529
00:22:56.355 --> 00:22:59.685
<00:22:56.355>basically <00:22:56.588>zero <00:22:56.821>cost <00:22:57.055>software <00:22:57.288>creation <00:22:57.521>for <00:22:57.755>everyone

530
00:22:59.865 --> 00:23:03.925
<00:22:59.865>for <00:23:00.145>very <00:23:00.425>small <00:23:00.705>personal <00:23:00.985>specific <00:23:01.265>applications.

531
00:23:04.085 --> 00:23:07.485
<00:23:04.085>I <00:23:04.225>don't <00:23:04.365>think <00:23:04.505>YouTube <00:23:04.645>will <00:23:04.785>be <00:23:04.925>wipe <00:23:05.065>coded <00:23:05.205>in <00:23:05.345>the <00:23:05.485>future.

532
00:23:07.595 --> 00:23:09.085
<00:23:07.595>Like <00:23:07.818>that's <00:23:08.042>probably <00:23:08.265>not <00:23:08.489>going <00:23:08.712>to <00:23:08.936>happen.

533
00:23:09.545 --> 00:23:10.965
<00:23:09.545>Uh, <00:23:09.971>I'm <00:23:10.397>100% <00:23:10.822>sure

534
00:23:10.965 --> 00:23:14.525
<00:23:10.965>that <00:23:11.120>we <00:23:11.276>will <00:23:11.431>get <00:23:11.587>more <00:23:11.742>optimized <00:23:11.898>hardwares <00:23:12.053>like <00:23:12.209>TPUs, <00:23:12.365>uh,

535
00:23:14.625 --> 00:23:17.805
<00:23:14.625>or <00:23:14.800>other <00:23:14.975>accelerators <00:23:15.150>which <00:23:15.325>will <00:23:15.500>drive <00:23:15.675>down <00:23:15.850>the <00:23:16.025>cost

536
00:23:17.825 --> 00:23:19.245
<00:23:17.825>for <00:23:18.144>using <00:23:18.464>those <00:23:18.783>AI <00:23:19.102>models.

537
00:23:19.705 --> 00:23:22.045
<00:23:19.705>Um, <00:23:19.985>the <00:23:20.264>importance <00:23:20.544>of <00:23:20.824>data <00:23:21.105>engineering

538
00:23:22.065 --> 00:23:23.365
<00:23:22.065>and <00:23:22.650>evaluation, <00:23:23.235>uh,

539
00:23:23.365 --> 00:23:25.325
<00:23:23.365>will <00:23:23.598>become <00:23:23.831>probably <00:23:24.065>the <00:23:24.298>most <00:23:24.531>crucial <00:23:24.765>skill

540
00:23:25.585 --> 00:23:27.565
<00:23:25.585>for <00:23:25.935>building <00:23:26.285>successful <00:23:26.635>AI <00:23:26.985>applications

541
00:23:27.565 --> 00:23:30.405
<00:23:27.565>because <00:23:27.765>the <00:23:27.965>bigger <00:23:28.165>we <00:23:28.365>scale <00:23:28.565>those <00:23:28.765>AP <00:23:28.965>applications,

542
00:23:30.425 --> 00:23:33.445
<00:23:30.425>the <00:23:30.565>more <00:23:30.705>complex <00:23:30.845>they <00:23:30.985>become, <00:23:31.125>the <00:23:31.264>more <00:23:31.405>important <00:23:31.544>it <00:23:31.685>will <00:23:31.825>be

543
00:23:33.835 --> 00:23:35.245
<00:23:33.835>that <00:23:34.152>we <00:23:34.469>can <00:23:34.786>evaluate <00:23:35.103>those

544
00:23:35.245 --> 00:23:38.685
<00:23:35.245>because <00:23:35.478>they're <00:23:35.711>very <00:23:35.945>different <00:23:36.178>from <00:23:36.411>software <00:23:36.645>development.

545
00:23:39.475 --> 00:23:42.005
<00:23:39.475>Then <00:23:39.649>I <00:23:39.824>believe <00:23:40.000>that <00:23:40.175>we <00:23:40.350>will <00:23:40.524>have <00:23:40.699>very <00:23:40.875>personalized

546
00:23:42.415 --> 00:23:45.005
<00:23:42.415>multimodal <00:23:42.695>outputs, <00:23:42.975>uh, <00:23:43.254>especially <00:23:43.534>for <00:23:43.815>entertainment.

547
00:23:45.545 --> 00:23:47.365
<00:23:45.545>And <00:23:45.778>maybe <00:23:46.011>even <00:23:46.245>we <00:23:46.478>will <00:23:46.711>see, <00:23:46.945>uh,

548
00:23:47.365 --> 00:23:49.565
<00:23:47.365>interfaces <00:23:47.715>which <00:23:48.065>are <00:23:48.415>completely <00:23:48.765>personalized.

549
00:23:49.585 --> 00:23:52.245
<00:23:49.585>So <00:23:49.740>maybe <00:23:49.896>my <00:23:50.051>Instagram <00:23:50.207>will <00:23:50.362>look <00:23:50.518>much <00:23:50.673>different <00:23:50.829>than <00:23:50.985>his

550
00:23:52.245 --> 00:23:54.245
<00:23:52.245>Instagram <00:23:52.594>or <00:23:52.945>TikTok <00:23:53.294>or <00:23:53.645>whatever.

551
00:23:54.245 --> 00:23:56.285
<00:23:54.245>We <00:23:54.478>already <00:23:54.711>see <00:23:54.945>some <00:23:55.178>indications <00:23:55.411>with <00:23:55.645>all

552
00:23:56.285 --> 00:23:58.205
<00:23:56.285>of <00:23:56.485>the <00:23:56.685>algorithms <00:23:56.885>on <00:23:57.085>what <00:23:57.285>content <00:23:57.485>we <00:23:57.685>see,

553
00:23:58.385 --> 00:24:00.565
<00:23:58.385>but <00:23:58.585>I <00:23:58.785>believe <00:23:58.985>the <00:23:59.185>interfaces <00:23:59.385>will <00:23:59.585>also <00:23:59.785>change

554
00:24:00.625 --> 00:24:02.365
<00:24:00.625>and <00:24:00.905>then <00:24:01.185>of <00:24:01.465>course, <00:24:01.745>uh, <00:24:02.025>videos

555
00:24:02.585 --> 00:24:05.845
<00:24:02.585>and <00:24:02.785>other <00:24:02.985>parts <00:24:03.185>will <00:24:03.385>be <00:24:03.585>much <00:24:03.785>more <00:24:03.985>personalized.

556
00:24:06.225 --> 00:24:09.165
<00:24:06.225>And <00:24:06.380>then <00:24:06.536>I <00:24:06.691>believe <00:24:06.847>that <00:24:07.002>with <00:24:07.158>all <00:24:07.313>of <00:24:07.469>those <00:24:07.625>agents

557
00:24:09.165 --> 00:24:11.245
<00:24:09.165>and <00:24:09.514>improvements <00:24:09.865>in <00:24:10.215>LLM <00:24:10.565>reasoning,

558
00:24:11.625 --> 00:24:14.405
<00:24:11.625>we <00:24:11.800>will <00:24:11.975>see <00:24:12.150>a <00:24:12.325>much <00:24:12.500>faster <00:24:12.675>acceleration <00:24:12.850>in <00:24:13.025>adoption

559
00:24:14.425 --> 00:24:17.565
<00:24:14.425>for <00:24:14.600>robotics <00:24:14.774>and <00:24:14.950>hopefully <00:24:15.125>soon <00:24:15.300>some <00:24:15.475>cool <00:24:15.649>humanoid <00:24:15.825>robotics

560
00:24:17.565 --> 00:24:18.605
<00:24:17.565>which <00:24:17.877>everyone <00:24:18.189>can <00:24:18.501>afford.