WEBVTT

1
00:00:02.200 --> 00:00:04.260
<00:00:02.200>Hello, <00:00:02.400>and <00:00:02.600>welcome <00:00:02.800>to <00:00:03.000>another <00:00:03.200>Coffee <00:00:03.400>With <00:00:03.600>Developers.

2
00:00:04.340 --> 00:00:08.260
<00:00:04.340>Today <00:00:04.467>I'm <00:00:04.594>joined <00:00:04.721>by <00:00:04.849>Alfonso <00:00:04.976>Graziano, <00:00:05.103>who <00:00:05.230>sent <00:00:05.358>in <00:00:05.485>a <00:00:05.612>paper <00:00:05.740>for

3
00:00:08.320 --> 00:00:12.080
<00:00:08.320>the <00:00:08.520>WeAreDevelopers <00:00:08.720>World <00:00:08.920>Congress <00:00:09.120>about <00:00:09.320>building <00:00:09.520>AI <00:00:09.720>agents

4
00:00:12.180 --> 00:00:15.920
<00:00:12.180>and <00:00:12.287>what <00:00:12.395>it <00:00:12.503>actually <00:00:12.610>means <00:00:12.718>to <00:00:12.826>do <00:00:12.933>that <00:00:13.041>in <00:00:13.149>real <00:00:13.256>life, <00:00:13.364>things <00:00:13.472>that <00:00:13.580>tutorials

5
00:00:15.980 --> 00:00:19.230
<00:00:15.980>don't <00:00:16.073>tell <00:00:16.166>you. <00:00:16.260>But <00:00:16.353>first <00:00:16.446>of <00:00:16.540>all, <00:00:16.633>let's <00:00:16.726>get <00:00:16.820>him <00:00:16.913>here <00:00:17.006>on <00:00:17.100>stage <00:00:17.193>and <00:00:17.286>get <00:00:17.380>him

6
00:00:19.260 --> 00:00:21.340
<00:00:19.260>invited. <00:00:19.540>How <00:00:19.820>are <00:00:20.100>you <00:00:20.380>doing, <00:00:20.660>Alfonso?

7
00:00:21.820 --> 00:00:24.880
<00:00:21.820>Hi, <00:00:21.936>everyone. <00:00:22.053>Yeah, <00:00:22.170>I'm <00:00:22.286>doing <00:00:22.403>great. <00:00:22.520>Thanks <00:00:22.636>a <00:00:22.753>lot, <00:00:22.870>Chris, <00:00:22.986>for <00:00:23.103>the <00:00:23.220>invite.

8
00:00:25.140 --> 00:00:25.599
<00:00:25.140>Appreciate <00:00:25.553>it.

9
00:00:25.800 --> 00:00:29.720
<00:00:25.800>Excellent. <00:00:25.907>So <00:00:26.015>you <00:00:26.123>work <00:00:26.230>at <00:00:26.338>NearForm, <00:00:26.446>which <00:00:26.553>means <00:00:26.661>you <00:00:26.769>do <00:00:26.876>a <00:00:26.984>lot <00:00:27.092>of <00:00:27.200>agent

10
00:00:29.780 --> 00:00:33.740
<00:00:29.780>work <00:00:29.935>every <00:00:30.091>day? <00:00:30.246>Or <00:00:30.402>how <00:00:30.557>did <00:00:30.713>you <00:00:30.868>pivot <00:00:31.024>into <00:00:31.180>that

11
00:00:34.400 --> 00:00:35.920
<00:00:34.400>field, <00:00:34.595>and <00:00:34.790>how <00:00:34.986>is <00:00:35.181>it <00:00:35.377>growing <00:00:35.572>right <00:00:35.768>now?

12
00:00:36.300 --> 00:00:39.860
<00:00:36.300>Yeah, <00:00:36.533>pretty <00:00:36.766>much. <00:00:37.000>So <00:00:37.233>actually, <00:00:37.466>NearForm <00:00:37.699>is

13
00:00:39.900 --> 00:00:41.440
<00:00:39.900>basically <00:00:40.361>a <00:00:40.824>consultancy <00:00:41.286>company.

14
00:00:41.580 --> 00:00:45.560
<00:00:41.580>We <00:00:41.687>do <00:00:41.795>a <00:00:41.903>lot <00:00:42.010>of <00:00:42.118>work <00:00:42.226>with <00:00:42.333>our <00:00:42.441>clients, <00:00:42.549>and <00:00:42.656>we <00:00:42.764>work <00:00:42.872>in <00:00:42.980>a

15
00:00:45.600 --> 00:00:48.260
<00:00:45.600>way <00:00:45.800>which <00:00:46.000>we <00:00:46.200>call <00:00:46.400>AI <00:00:46.600>native <00:00:46.800>engineering, <00:00:47.000>right?

16
00:00:48.320 --> 00:00:50.100
<00:00:48.320>So <00:00:48.786>basically, <00:00:49.253>our <00:00:49.720>engineers

17
00:00:50.960 --> 00:00:54.000
<00:00:50.960>use <00:00:51.193>heavily <00:00:51.426>AI <00:00:51.660>during <00:00:51.893>our <00:00:52.126>day-to-day <00:00:52.360>job

18
00:00:55.220 --> 00:00:58.420
<00:00:55.220>in <00:00:55.360>a <00:00:55.500>professional <00:00:55.640>way, <00:00:55.780>of <00:00:55.920>course. <00:00:56.060>We <00:00:56.199>don't <00:00:56.339>vibe <00:00:56.480>code, <00:00:56.620>hopefully.

19
00:00:59.200 --> 00:01:03.160
<00:00:59.200>And <00:00:59.340>a <00:00:59.480>lot <00:00:59.620>of <00:00:59.760>those <00:00:59.900>gigs <00:01:00.040>these <00:01:00.180>days <00:01:00.320>are <00:01:00.460>related <00:01:00.600>to

20
00:01:03.320 --> 00:01:07.220
<00:01:03.320>building <00:01:03.475>AI <00:01:03.631>agents <00:01:03.786>for <00:01:03.942>our <00:01:04.097>clients, <00:01:04.253>building <00:01:04.408>our <00:01:04.564>AI <00:01:04.720>agents

21
00:01:07.260 --> 00:01:11.240
<00:01:07.260>internally, <00:01:07.387>which <00:01:07.514>we <00:01:07.641>use <00:01:07.769>for <00:01:07.896>internal <00:01:08.023>use <00:01:08.150>as <00:01:08.278>well, <00:01:08.405>which <00:01:08.532>is <00:01:08.660>more

22
00:01:11.260 --> 00:01:14.660
<00:01:11.260>interesting <00:01:11.400>than <00:01:11.540>it <00:01:11.680>looks <00:01:11.820>like. <00:01:11.960>But <00:01:12.100>I'm <00:01:12.240>currently <00:01:12.380>working <00:01:12.520>on <00:01:12.660>a

23
00:01:14.700 --> 00:01:16.259
<00:01:14.700>client-facing <00:01:15.400>AI <00:01:16.100>agent.

24
00:01:17.460 --> 00:01:21.310
<00:01:17.460>And <00:01:17.587>who <00:01:17.714>are <00:01:17.841>your <00:01:17.969>clients? <00:01:18.096>Do <00:01:18.223>you <00:01:18.350>have <00:01:18.478>end <00:01:18.605>users, <00:01:18.732>or <00:01:18.860>do

25
00:01:21.400 --> 00:01:25.140
<00:01:21.400>you <00:01:21.516>have <00:01:21.633>actually <00:01:21.750>other <00:01:21.866>tech <00:01:21.983>engineering <00:01:22.100>teams <00:01:22.216>that <00:01:22.333>want <00:01:22.450>to <00:01:22.566>get <00:01:22.683>into <00:01:22.800>agentic

26
00:01:25.160 --> 00:01:25.720
development?

27
00:01:26.340 --> 00:01:28.740
<00:01:26.340>Yeah, <00:01:26.573>pretty <00:01:26.806>much <00:01:27.040>it's <00:01:27.273>other <00:01:27.506>engineering <00:01:27.740>teams.

28
00:01:30.140 --> 00:01:30.480
<00:01:30.140>Of <00:01:30.446>course,

29
00:01:31.440 --> 00:01:34.500
<00:01:31.440>some <00:01:31.615>of <00:01:31.789>the <00:01:31.965>agents <00:01:32.140>that <00:01:32.315>we <00:01:32.490>build <00:01:32.664>are <00:01:32.840>for

30
00:01:34.840 --> 00:01:38.679
<00:01:34.840>end <00:01:35.040>users <00:01:35.240>from <00:01:35.440>our <00:01:35.640>clients. <00:01:35.840>Some <00:01:36.040>of <00:01:36.240>them

31
00:01:38.740 --> 00:01:41.680
<00:01:38.740>are <00:01:38.880>for <00:01:39.020>internal <00:01:39.160>use. <00:01:39.300>So <00:01:39.440>it's <00:01:39.580>a <00:01:39.720>kind <00:01:39.860>of <00:01:40.000>a <00:01:40.140>mix.

32
00:01:41.900 --> 00:01:45.510
<00:01:41.900>But <00:01:42.000>right <00:01:42.100>now, <00:01:42.200>I'm <00:01:42.300>working <00:01:42.400>on <00:01:42.500>an <00:01:42.600>agent <00:01:42.700>which <00:01:42.800>is <00:01:42.900>going <00:01:43.000>to <00:01:43.100>be <00:01:43.200>used <00:01:43.300>by

33
00:01:46.320 --> 00:01:50.260
<00:01:46.320>the <00:01:46.520>final <00:01:46.720>user, <00:01:46.919>right, <00:01:47.119>so <00:01:47.320>the <00:01:47.520>final <00:01:47.720>customers.

34
00:01:50.690 --> 00:01:54.520
<00:01:50.690>So <00:01:50.865>in <00:01:51.039>your <00:01:51.215>description <00:01:51.390>of <00:01:51.565>your <00:01:51.740>talk <00:01:51.914>you <00:01:52.090>wrote

35
00:01:54.600 --> 00:01:58.540
<00:01:54.600>about <00:01:54.699>that <00:01:54.800>the <00:01:54.899>biggest <00:01:55.000>problem <00:01:55.100>that <00:01:55.199>you <00:01:55.300>see <00:01:55.399>is <00:01:55.500>that <00:01:55.600>building <00:01:55.699>an <00:01:55.800>agent <00:01:55.899>is <00:01:56.000>not

36
00:01:58.580 --> 00:02:02.220
<00:01:58.580>that <00:01:58.673>much <00:01:58.766>of <00:01:58.860>an <00:01:58.953>issue, <00:01:59.046>but <00:01:59.140>you <00:01:59.233>might <00:01:59.326>be <00:01:59.420>in <00:01:59.513>a <00:01:59.606>highly <00:01:59.700>regulated <00:01:59.793>environment <00:01:59.886>where <00:01:59.980>you

37
00:02:02.280 --> 00:02:03.460
<00:02:02.280>actually <00:02:02.545>have <00:02:02.811>to <00:02:03.076>think <00:02:03.342>about

38
00:02:04.560 --> 00:02:08.150
<00:02:04.560>safeguards, <00:02:04.735>guardrails, <00:02:04.910>all <00:02:05.085>kind <00:02:05.260>of <00:02:05.435>ways <00:02:05.610>of <00:02:05.785>rolling <00:02:05.960>things

39
00:02:08.280 --> 00:02:12.160
<00:02:08.280>out. <00:02:08.387>What <00:02:08.495>are <00:02:08.603>the <00:02:08.710>things <00:02:08.818>that <00:02:08.926>you <00:02:09.033>encountered <00:02:09.141>where <00:02:09.249>most <00:02:09.356>people <00:02:09.464>that <00:02:09.572>want <00:02:09.680>to

40
00:02:12.200 --> 00:02:14.150
<00:02:12.200>get <00:02:12.480>into <00:02:12.760>building <00:02:13.040>agents <00:02:13.320>get <00:02:13.600>stuck?

41
00:02:14.500 --> 00:02:15.000
<00:02:14.500>Yeah, <00:02:14.950>so

42
00:02:16.020 --> 00:02:18.030
<00:02:16.020>we <00:02:16.220>can <00:02:16.420>speak <00:02:16.620>about <00:02:16.820>that <00:02:17.020>for <00:02:17.220>hours. <00:02:17.420>But

43
00:02:18.900 --> 00:02:22.580
<00:02:18.900>just <00:02:18.993>to <00:02:19.086>summarize, <00:02:19.180>the <00:02:19.273>first <00:02:19.366>thing <00:02:19.460>that <00:02:19.553>you <00:02:19.646>do <00:02:19.740>when <00:02:19.833>you're <00:02:19.926>trying <00:02:20.020>to <00:02:20.113>build <00:02:20.206>an <00:02:20.300>agent

44
00:02:22.700 --> 00:02:26.350
<00:02:22.700>is-- <00:02:22.855>Just <00:02:23.011>to <00:02:23.166>summarize <00:02:23.322>for <00:02:23.477>the <00:02:23.633>audience, <00:02:23.788>an <00:02:23.944>agent <00:02:24.100>is

45
00:02:26.380 --> 00:02:29.760
<00:02:26.380>basically <00:02:26.579>an <00:02:26.780>agentic <00:02:26.980>loop, <00:02:27.180>which <00:02:27.380>is <00:02:27.579>using <00:02:27.780>an

46
00:02:29.860 --> 00:02:33.520
<00:02:29.860>LLM <00:02:30.035>within <00:02:30.210>an <00:02:30.385>agentic <00:02:30.560>loop, <00:02:30.735>which <00:02:30.910>is <00:02:31.085>like <00:02:31.260>calling

47
00:02:33.580 --> 00:02:36.480
<00:02:33.579>tools. <00:02:33.859>It <00:02:34.140>has <00:02:34.420>some <00:02:34.700>memory <00:02:34.980>sometimes.

48
00:02:36.540 --> 00:02:40.340
<00:02:36.540>So <00:02:36.773>an <00:02:37.006>AI <00:02:37.239>agent <00:02:37.473>is <00:02:37.706>basically <00:02:37.940>a

49
00:02:40.400 --> 00:02:43.440
<00:02:40.400>system <00:02:40.575>by <00:02:40.750>itself. <00:02:40.925>There <00:02:41.100>are <00:02:41.275>multiple <00:02:41.450>pieces, <00:02:41.625>and <00:02:41.800>the

50
00:02:45.000 --> 00:02:48.380
<00:02:45.000>LLM <00:02:45.140>is <00:02:45.280>one <00:02:45.420>of <00:02:45.560>the <00:02:45.700>core <00:02:45.840>pieces <00:02:45.980>of <00:02:46.120>the <00:02:46.260>entire <00:02:46.400>system.

51
00:02:48.780 --> 00:02:52.000
<00:02:48.780>But <00:02:48.873>then <00:02:48.966>there <00:02:49.060>are <00:02:49.153>a <00:02:49.246>lot <00:02:49.340>of <00:02:49.433>other <00:02:49.526>things <00:02:49.620>that <00:02:49.713>we <00:02:49.806>have <00:02:49.900>to <00:02:49.993>take <00:02:50.086>into <00:02:50.180>account.

52
00:02:52.640 --> 00:02:56.480
<00:02:52.640>Now, <00:02:52.727>when <00:02:52.815>we <00:02:52.902>start <00:02:52.989>for <00:02:53.077>the <00:02:53.165>first <00:02:53.252>time <00:02:53.339>to <00:02:53.427>build <00:02:53.515>an <00:02:53.602>agent, <00:02:53.690>the <00:02:53.777>first <00:02:53.864>thing <00:02:53.952>that <00:02:54.040>we

53
00:02:56.500 --> 00:02:59.480
<00:02:56.500>do <00:02:56.700>is <00:02:56.900>we <00:02:57.100>pick <00:02:57.300>a <00:02:57.500>framework <00:02:57.700>of <00:02:57.900>choice.

54
00:02:59.690 --> 00:03:02.440
<00:02:59.690>Might <00:02:59.970>be <00:03:00.250>LangChain, <00:03:00.530>might <00:03:00.810>be <00:03:01.090>Mastra.

55
00:03:02.480 --> 00:03:05.580
<00:03:02.480>It <00:03:02.619>depends <00:03:02.760>a <00:03:02.899>little <00:03:03.040>bit <00:03:03.179>on <00:03:03.320>your <00:03:03.459>technology <00:03:03.600>stack <00:03:03.739>as <00:03:03.880>well.

56
00:03:06.040 --> 00:03:08.580
<00:03:06.040>Then <00:03:06.239>we <00:03:06.440>build <00:03:06.640>a <00:03:06.840>system <00:03:07.040>prompt <00:03:07.239>of <00:03:07.440>something.

57
00:03:08.620 --> 00:03:11.800
<00:03:08.620>We <00:03:08.727>try <00:03:08.835>to <00:03:08.943>add <00:03:09.050>a <00:03:09.158>couple <00:03:09.266>of <00:03:09.373>tools, <00:03:09.481>and <00:03:09.589>it <00:03:09.696>kind <00:03:09.804>of <00:03:09.912>looks <00:03:10.020>good.

58
00:03:12.300 --> 00:03:16.100
<00:03:12.300>If <00:03:12.407>you <00:03:12.515>try <00:03:12.623>it <00:03:12.730>for <00:03:12.838>a <00:03:12.946>couple <00:03:13.053>of <00:03:13.161>times, <00:03:13.269>it <00:03:13.376>seems <00:03:13.484>that <00:03:13.592>it's <00:03:13.700>working,

59
00:03:16.140 --> 00:03:18.800
<00:03:16.140>it <00:03:16.315>seems <00:03:16.489>that <00:03:16.665>it's <00:03:16.839>doing <00:03:17.015>what <00:03:17.190>it <00:03:17.364>should <00:03:17.540>do.

60
00:03:20.060 --> 00:03:23.900
<00:03:20.060>But <00:03:20.142>then <00:03:20.224>you <00:03:20.307>try <00:03:20.389>to <00:03:20.471>put <00:03:20.554>it <00:03:20.636>in <00:03:20.718>the <00:03:20.801>hands <00:03:20.883>of <00:03:20.965>customers, <00:03:21.048>you <00:03:21.130>get <00:03:21.212>a <00:03:21.295>lot <00:03:21.377>of <00:03:21.460>negative

61
00:03:23.960 --> 00:03:27.400
<00:03:23.960>feedback, <00:03:24.115>let's <00:03:24.271>put <00:03:24.426>it <00:03:24.582>this <00:03:24.737>way. <00:03:24.893>Or <00:03:25.048>it's <00:03:25.204>not <00:03:25.360>following

62
00:03:27.980 --> 00:03:29.420
<00:03:27.980>80% <00:03:28.239>of <00:03:28.498>what <00:03:28.757>you <00:03:29.016>were <00:03:29.276>expecting.

63
00:03:30.880 --> 00:03:34.220
<00:03:30.880>And <00:03:31.055>that's <00:03:31.230>because <00:03:31.405>you <00:03:31.579>didn't <00:03:31.755>build <00:03:31.930>a <00:03:32.105>proper <00:03:32.280>system.

64
00:03:34.700 --> 00:03:38.640
<00:03:34.700>You <00:03:34.799>just <00:03:34.899>built <00:03:35.000>a <00:03:35.100>set <00:03:35.200>of <00:03:35.299>prompts <00:03:35.399>which <00:03:35.500>are <00:03:35.600>going <00:03:35.700>to <00:03:35.799>answer <00:03:35.899>a <00:03:36.000>couple <00:03:36.100>of

65
00:03:39.040 --> 00:03:41.540
<00:03:39.040>very <00:03:39.273>narrow <00:03:39.506>questions, <00:03:39.739>hopefully <00:03:39.973>from <00:03:40.206>your <00:03:40.440>customers.

66
00:03:42.000 --> 00:03:42.780
<00:03:42.000>But <00:03:42.351>when <00:03:42.702>we

67
00:03:43.840 --> 00:03:45.680
<00:03:43.840>use <00:03:44.120>natural <00:03:44.400>language <00:03:44.680>as <00:03:44.960>an <00:03:45.240>interface,

68
00:03:46.580 --> 00:03:50.240
<00:03:46.579>we <00:03:46.662>should <00:03:46.744>be <00:03:46.827>prepared <00:03:46.909>to <00:03:46.991>the <00:03:47.074>fact <00:03:47.156>that <00:03:47.238>our <00:03:47.321>users <00:03:47.403>are <00:03:47.485>going <00:03:47.568>to <00:03:47.650>ask <00:03:47.732>us <00:03:47.815>a <00:03:47.897>lot <00:03:47.980>of

69
00:03:50.300 --> 00:03:53.990
<00:03:50.300>different <00:03:50.416>kinds <00:03:50.533>of <00:03:50.650>questions, <00:03:50.766>and <00:03:50.883>we <00:03:51.000>should <00:03:51.116>be <00:03:51.233>prepared <00:03:51.350>to <00:03:51.466>answer <00:03:51.583>all <00:03:51.700>those

70
00:03:54.020 --> 00:03:57.780
<00:03:54.020>questions, <00:03:54.220>and <00:03:54.420>the <00:03:54.620>system <00:03:54.820>should <00:03:55.020>be <00:03:55.220>reliable <00:03:55.420>enough.

71
00:03:58.140 --> 00:04:01.500
<00:03:58.140>Which <00:03:58.295>of <00:03:58.451>course, <00:03:58.606>given <00:03:58.762>that <00:03:58.917>LLMs <00:03:59.073>are <00:03:59.228>not <00:03:59.384>deterministic, <00:03:59.540>it's

72
00:04:02.000 --> 00:04:05.040
<00:04:02.000>super <00:04:02.140>complex, <00:04:02.280>and <00:04:02.420>it <00:04:02.560>has <00:04:02.700>a <00:04:02.840>lot <00:04:02.980>of <00:04:03.120>very <00:04:03.260>interesting <00:04:03.400>challenges.

73
00:04:05.820 --> 00:04:09.560
<00:04:05.820>It <00:04:05.913>feels <00:04:06.006>like <00:04:06.100>back <00:04:06.193>in <00:04:06.286>the <00:04:06.380>days <00:04:06.473>when <00:04:06.566>you <00:04:06.660>build <00:04:06.753>a <00:04:06.846>system, <00:04:06.940>you <00:04:07.033>always <00:04:07.126>got <00:04:07.220>a

74
00:04:09.570 --> 00:04:13.560
<00:04:09.570>database, <00:04:09.686>and <00:04:09.803>you <00:04:09.920>got <00:04:10.036>an <00:04:10.153>environment <00:04:10.269>that <00:04:10.386>actually <00:04:10.503>has <00:04:10.620>all <00:04:10.736>the <00:04:10.853>happy <00:04:10.970>path,

75
00:04:13.640 --> 00:04:17.149
<00:04:13.640>has <00:04:13.815>all <00:04:13.989>the <00:04:14.165>right <00:04:14.339>things <00:04:14.515>in <00:04:14.690>there. <00:04:14.864>Like <00:04:15.040>every

76
00:04:17.221 --> 00:04:20.721
<00:04:17.221>tutorial <00:04:17.321>out <00:04:17.421>there <00:04:17.521>is <00:04:17.620>like, <00:04:17.721>"Build <00:04:17.821>your <00:04:17.921>agent, <00:04:18.021>and <00:04:18.120>here <00:04:18.221>you <00:04:18.321>get <00:04:18.421>the <00:04:18.521>first <00:04:18.620>answers,"

77
00:04:20.780 --> 00:04:24.130
<00:04:20.779>or, <00:04:20.873>"Here <00:04:20.966>it <00:04:21.059>creates <00:04:21.153>images <00:04:21.246>for <00:04:21.339>you," <00:04:21.433>or, <00:04:21.526>"Here <00:04:21.619>it <00:04:21.713>creates <00:04:21.806>a <00:04:21.899>video <00:04:21.993>for <00:04:22.086>you," <00:04:22.179>or,

78
00:04:24.190 --> 00:04:27.940
<00:04:24.190>"Does <00:04:24.330>the <00:04:24.469>research <00:04:24.610>for <00:04:24.750>you <00:04:24.890>and <00:04:25.030>compares <00:04:25.170>different <00:04:25.310>resources <00:04:25.450>for <00:04:25.590>you."

79
00:04:28.620 --> 00:04:30.340
<00:04:28.620>And <00:04:29.086>I <00:04:29.553>think <00:04:30.020>the

80
00:04:31.160 --> 00:04:35.120
<00:04:31.160>problem <00:04:31.287>that <00:04:31.414>these <00:04:31.541>tutorials <00:04:31.669>have <00:04:31.796>is <00:04:31.923>that <00:04:32.050>they <00:04:32.178>show <00:04:32.305>you <00:04:32.432>only <00:04:32.560>the

81
00:04:35.200 --> 00:04:37.860
<00:04:35.200>thing <00:04:35.355>that <00:04:35.511>works <00:04:35.666>rather <00:04:35.822>than <00:04:35.977>the <00:04:36.133>thing <00:04:36.288>that <00:04:36.444>doesn't <00:04:36.599>work.

82
00:04:38.460 --> 00:04:42.280
<00:04:38.460>Now, <00:04:38.599>how <00:04:38.739>would <00:04:38.880>you <00:04:39.020>do <00:04:39.159>that <00:04:39.299>in <00:04:39.440>terms <00:04:39.580>of <00:04:39.719>like <00:04:39.859>as

83
00:04:42.740 --> 00:04:46.420
<00:04:42.740>somebody <00:04:42.867>who <00:04:42.994>just <00:04:43.121>wants <00:04:43.249>to <00:04:43.376>build <00:04:43.503>their <00:04:43.630>first <00:04:43.758>agent, <00:04:43.885>what <00:04:44.012>would <00:04:44.140>you

84
00:04:46.540 --> 00:04:50.340
<00:04:46.540>say <00:04:46.680>are <00:04:46.820>danger <00:04:46.960>signs <00:04:47.100>in <00:04:47.240>tutorials <00:04:47.380>that <00:04:47.520>actually <00:04:47.660>people <00:04:47.800>should <00:04:47.940>look

85
00:04:50.440 --> 00:04:50.640
for?

86
00:04:51.900 --> 00:04:55.560
<00:04:51.900>Yeah. <00:04:52.179>So <00:04:52.460>tutorials <00:04:52.739>that <00:04:53.020>make <00:04:53.299>things

87
00:04:56.040 --> 00:04:58.000
<00:04:56.040>look <00:04:56.320>too <00:04:56.600>simplistic, <00:04:56.880>I <00:04:57.160>would <00:04:57.440>say.

88
00:04:59.020 --> 00:05:00.100
<00:04:59.020>Because <00:04:59.263>one <00:04:59.506>thing <00:04:59.749>is <00:04:59.992>we

89
00:05:01.680 --> 00:05:05.340
<00:05:01.680>all <00:05:01.807>have <00:05:01.934>been <00:05:02.061>there, <00:05:02.189>right? <00:05:02.316>So <00:05:02.443>we <00:05:02.570>all <00:05:02.698>have <00:05:02.825>looked <00:05:02.952>at <00:05:03.080>the

90
00:05:05.640 --> 00:05:09.560
<00:05:05.640>React <00:05:05.747>tutorial <00:05:05.855>or <00:05:05.963>something. <00:05:06.070>We <00:05:06.178>look <00:05:06.286>at <00:05:06.393>the <00:05:06.501>tutorial, <00:05:06.609>it <00:05:06.716>feels <00:05:06.824>good <00:05:06.932>because <00:05:07.039>we

91
00:05:09.600 --> 00:05:13.580
<00:05:09.600>can <00:05:09.755>reproduce <00:05:09.911>what <00:05:10.066>we <00:05:10.222>do <00:05:10.377>have. <00:05:10.533>But <00:05:10.688>then <00:05:10.844>productionize <00:05:11.000>something,

92
00:05:13.800 --> 00:05:17.760
<00:05:13.800>it's <00:05:13.975>completely <00:05:14.150>different. <00:05:14.325>So <00:05:14.500>a <00:05:14.675>lot <00:05:14.850>of <00:05:15.025>times, <00:05:15.200>like

93
00:05:18.040 --> 00:05:21.760
<00:05:18.040>Agents <00:05:18.195>101 <00:05:18.351>tutorial, <00:05:18.506>these <00:05:18.662>kind <00:05:18.817>of <00:05:18.973>tutorials, <00:05:19.128>they <00:05:19.284>don't <00:05:19.440>think

94
00:05:21.820 --> 00:05:25.740
<00:05:21.820>about <00:05:21.975>evals. <00:05:22.131>They <00:05:22.286>don't <00:05:22.442>discuss <00:05:22.597>what <00:05:22.753>is <00:05:22.908>golden <00:05:23.064>dataset, <00:05:23.219>which

95
00:05:25.880 --> 00:05:29.840
<00:05:25.880>is <00:05:26.113>incredibly <00:05:26.346>difficult <00:05:26.580>to <00:05:26.813>build <00:05:27.046>sometimes, <00:05:27.280>a

96
00:05:29.860 --> 00:05:33.580
<00:05:29.860>proper <00:05:30.060>golden <00:05:30.260>dataset. <00:05:30.460>Building <00:05:30.660>proper <00:05:30.860>evals, <00:05:31.060>having <00:05:31.260>proper

97
00:05:33.680 --> 00:05:35.320
<00:05:33.680>tracing <00:05:34.146>of <00:05:34.613>what's <00:05:35.080>happening,

98
00:05:36.140 --> 00:05:40.020
<00:05:36.140>getting <00:05:36.373>feedback <00:05:36.606>live <00:05:36.840>from <00:05:37.073>the <00:05:37.306>users, <00:05:37.539>evaluating

99
00:05:40.060 --> 00:05:43.780
<00:05:40.060>that <00:05:40.200>feedback, <00:05:40.340>and <00:05:40.480>having <00:05:40.620>a <00:05:40.760>feedback <00:05:40.900>loop <00:05:41.040>so <00:05:41.180>that <00:05:41.320>we <00:05:41.460>can

100
00:05:43.840 --> 00:05:47.160
<00:05:43.840>use <00:05:44.073>user <00:05:44.306>informations <00:05:44.540>to <00:05:44.773>harden <00:05:45.006>the <00:05:45.240>system.

101
00:05:47.720 --> 00:05:49.900
<00:05:47.720>Then <00:05:48.186>things <00:05:48.653>like <00:05:49.120>guardrails,

102
00:05:51.720 --> 00:05:55.420
<00:05:51.720>things <00:05:51.860>like <00:05:52.000>security, <00:05:52.140>of <00:05:52.280>course. <00:05:52.420>So <00:05:52.560>what <00:05:52.700>we <00:05:52.840>want <00:05:52.980>is <00:05:53.120>to

103
00:05:55.460 --> 00:05:59.200
<00:05:55.460>ensure <00:05:55.615>that <00:05:55.771>our <00:05:55.926>AI <00:05:56.082>agent <00:05:56.237>has <00:05:56.393>access <00:05:56.548>to <00:05:56.704>enough <00:05:56.859>tools,

104
00:06:00.200 --> 00:06:03.640
<00:06:00.200>but <00:06:00.300>has <00:06:00.400>access <00:06:00.500>to <00:06:00.599>those <00:06:00.700>tools <00:06:00.800>with <00:06:00.900>the <00:06:01.000>right <00:06:01.099>level <00:06:01.200>of <00:06:01.300>permission <00:06:01.400>of <00:06:01.500>the <00:06:01.599>user,

105
00:06:04.040 --> 00:06:04.160
right?

106
00:06:05.180 --> 00:06:09.030
<00:06:05.180>Let's <00:06:05.296>say <00:06:05.413>that <00:06:05.530>I <00:06:05.646>have <00:06:05.763>an <00:06:05.880>AI <00:06:05.996>agent <00:06:06.113>working <00:06:06.230>in <00:06:06.346>my, <00:06:06.463>I <00:06:06.580>don't

107
00:06:09.060 --> 00:06:10.910
<00:06:09.060>know... <00:06:09.526>company <00:06:09.993>knowledge <00:06:10.460>base.

108
00:06:12.030 --> 00:06:15.930
<00:06:12.030>If <00:06:12.130>I <00:06:12.229>give <00:06:12.330>my <00:06:12.429>AI <00:06:12.530>access <00:06:12.630>to <00:06:12.729>the <00:06:12.830>entire <00:06:12.929>knowledge <00:06:13.030>base, <00:06:13.130>every <00:06:13.229>user <00:06:13.330>of <00:06:13.429>that

109
00:06:15.970 --> 00:06:19.830
<00:06:15.970>system, <00:06:16.110>potentially, <00:06:16.250>it's <00:06:16.390>going <00:06:16.530>to <00:06:16.670>have <00:06:16.810>access <00:06:16.950>to <00:06:17.090>the <00:06:17.230>entire <00:06:17.370>knowledge

110
00:06:19.850 --> 00:06:23.600
<00:06:19.850>base. <00:06:20.005>So <00:06:20.161>there <00:06:20.316>are <00:06:20.472>a <00:06:20.627>lot <00:06:20.783>of <00:06:20.938>things <00:06:21.094>that <00:06:21.250>people

111
00:06:23.630 --> 00:06:26.830
<00:06:23.630>forget <00:06:23.785>about <00:06:23.941>or, <00:06:24.096>well, <00:06:24.252>are <00:06:24.407>not <00:06:24.563>thinking <00:06:24.718>too <00:06:24.874>much <00:06:25.030>about.

112
00:06:27.550 --> 00:06:31.530
<00:06:27.550>And <00:06:27.690>in <00:06:27.830>fact, <00:06:27.970>there <00:06:28.110>is <00:06:28.250>also <00:06:28.390>the <00:06:28.530>OWASP <00:06:28.670>Top <00:06:28.810>10 <00:06:28.950>for

113
00:06:31.540 --> 00:06:34.710
<00:06:31.540>agentic <00:06:31.740>applications, <00:06:31.940>which <00:06:32.140>is, <00:06:32.340>I <00:06:32.540>believe, <00:06:32.740>super <00:06:32.940>important.

114
00:06:34.750 --> 00:06:38.370
<00:06:34.750>And <00:06:34.877>a <00:06:35.004>lot <00:06:35.131>of <00:06:35.259>people <00:06:35.386>are <00:06:35.513>skipping <00:06:35.640>security <00:06:35.768>as, <00:06:35.895>yeah, <00:06:36.022>that's <00:06:36.150>an

115
00:06:38.430 --> 00:06:40.330
<00:06:38.430>agent, <00:06:38.630>it's <00:06:38.830>going <00:06:39.030>to <00:06:39.230>figure <00:06:39.430>it <00:06:39.630>out. <00:06:39.830>No.

116
00:06:40.890 --> 00:06:44.530
<00:06:40.890>Security <00:06:41.030>should <00:06:41.169>be <00:06:41.310>one <00:06:41.450>of <00:06:41.590>your <00:06:41.729>first <00:06:41.870>concerns <00:06:42.010>in <00:06:42.150>this <00:06:42.289>case.

117
00:06:46.110 --> 00:06:49.730
<00:06:46.110>OWASP <00:06:46.217>has <00:06:46.325>been <00:06:46.433>around <00:06:46.540>for <00:06:46.648>a <00:06:46.756>long, <00:06:46.863>long <00:06:46.971>time, <00:06:47.079>and <00:06:47.186>for <00:06:47.294>all <00:06:47.402>different <00:06:47.510>environments.

118
00:06:49.780 --> 00:06:53.330
<00:06:49.780>I'm <00:06:49.896>actually <00:06:50.013>excited <00:06:50.130>to <00:06:50.246>hear <00:06:50.363>that <00:06:50.479>there's <00:06:50.596>already <00:06:50.713>a <00:06:50.830>Top <00:06:50.946>10 <00:06:51.063>for <00:06:51.179>agentic

119
00:06:53.390 --> 00:06:57.290
<00:06:53.390>environments. <00:06:53.545>Is <00:06:53.701>this <00:06:53.856>just <00:06:54.012>something <00:06:54.167>that <00:06:54.323>has <00:06:54.478>been <00:06:54.634>adapted <00:06:54.789>from

120
00:06:57.950 --> 00:07:00.860
<00:06:57.950>bot <00:06:58.066>environments, <00:06:58.183>or <00:06:58.300>is <00:06:58.416>this <00:06:58.533>something <00:06:58.650>that <00:06:58.766>has <00:06:58.883>been <00:06:59.000>written <00:06:59.116>completely <00:06:59.233>new <00:06:59.349>or

121
00:07:00.930 --> 00:07:01.830
<00:07:00.930>found <00:07:01.335>completely <00:07:01.740>new?

122
00:07:02.550 --> 00:07:06.490
<00:07:02.550>So <00:07:02.677>some <00:07:02.804>of <00:07:02.931>the <00:07:03.059>key <00:07:03.186>principles, <00:07:03.313>I <00:07:03.440>would <00:07:03.568>say <00:07:03.695>that <00:07:03.822>they <00:07:03.950>are

123
00:07:06.910 --> 00:07:09.370
<00:07:06.910>relatively <00:07:07.110>similar, <00:07:07.310>but <00:07:07.510>then <00:07:07.710>there <00:07:07.910>are <00:07:08.110>new <00:07:08.310>things.

124
00:07:09.670 --> 00:07:13.630
<00:07:09.670>Given <00:07:09.797>that <00:07:09.924>we <00:07:10.051>are <00:07:10.179>working <00:07:10.306>with <00:07:10.433>LLMs <00:07:10.560>and <00:07:10.688>non-deterministic <00:07:10.815>stuff, <00:07:10.942>there <00:07:11.070>are,

125
00:07:13.730 --> 00:07:14.330
<00:07:13.730>for <00:07:14.270>example,

126
00:07:15.310 --> 00:07:18.350
<00:07:15.310>prompt <00:07:15.590>injection <00:07:15.870>or <00:07:16.150>indirect <00:07:16.430>prompt <00:07:16.710>injection.

127
00:07:18.430 --> 00:07:22.350
<00:07:18.430>So <00:07:18.557>there <00:07:18.684>are <00:07:18.811>all <00:07:18.939>these <00:07:19.066>things <00:07:19.193>which <00:07:19.320>you <00:07:19.448>only <00:07:19.575>encounter <00:07:19.702>when <00:07:19.830>you're

128
00:07:22.390 --> 00:07:23.130
<00:07:22.390>working <00:07:23.056>with,

129
00:07:23.930 --> 00:07:27.710
<00:07:23.930>A, <00:07:24.105>a <00:07:24.280>non-deterministic <00:07:24.455>system <00:07:24.630>like <00:07:24.805>an <00:07:24.980>LLM, <00:07:25.155>and <00:07:25.330>B,

130
00:07:27.890 --> 00:07:31.630
<00:07:27.890>a <00:07:28.090>system <00:07:28.289>which <00:07:28.490>can <00:07:28.690>be <00:07:28.890>tricked <00:07:29.090>into <00:07:29.289>doing

131
00:07:31.770 --> 00:07:33.870
<00:07:31.770>something <00:07:32.003>that <00:07:32.236>it's <00:07:32.469>not <00:07:32.703>programmed <00:07:32.936>to <00:07:33.169>do.

132
00:07:34.750 --> 00:07:37.770
<00:07:34.750>Because <00:07:34.925>let's <00:07:35.100>say <00:07:35.275>that <00:07:35.450>I'm <00:07:35.625>using <00:07:35.800>a <00:07:35.975>normal <00:07:36.150>application.

133
00:07:38.370 --> 00:07:41.990
<00:07:38.370>I <00:07:38.477>can <00:07:38.585>try <00:07:38.693>to <00:07:38.800>hack <00:07:38.908>that <00:07:39.016>application, <00:07:39.123>but <00:07:39.231>I <00:07:39.339>cannot <00:07:39.446>try <00:07:39.554>to <00:07:39.662>trick <00:07:39.770>the

134
00:07:42.050 --> 00:07:45.450
<00:07:42.050>application <00:07:42.190>to <00:07:42.330>do <00:07:42.470>something <00:07:42.610>that <00:07:42.750>it <00:07:42.890>shouldn't <00:07:43.030>do. <00:07:43.170>Because <00:07:43.310>it's <00:07:43.450>deterministic.

135
00:07:46.030 --> 00:07:49.990
<00:07:46.030>In <00:07:46.137>the <00:07:46.245>case <00:07:46.353>of <00:07:46.460>an <00:07:46.568>AI <00:07:46.676>agent, <00:07:46.783>I <00:07:46.891>can <00:07:46.999>try <00:07:47.106>to <00:07:47.214>trick <00:07:47.322>the <00:07:47.429>system

136
00:07:50.130 --> 00:07:51.810
<00:07:50.130>in <00:07:50.409>a <00:07:50.690>lot <00:07:50.969>of <00:07:51.250>different <00:07:51.530>ways.

137
00:07:52.650 --> 00:07:56.330
<00:07:52.650>So <00:07:52.789>there <00:07:52.929>are <00:07:53.070>a <00:07:53.210>lot <00:07:53.349>of <00:07:53.489>new <00:07:53.630>challenges <00:07:53.770>with <00:07:53.909>AI <00:07:54.049>systems.

138
00:07:56.390 --> 00:07:59.690
<00:07:56.390>And <00:07:56.565>yeah, <00:07:56.740>the <00:07:56.914>OWASP <00:07:57.090>Top <00:07:57.265>10, <00:07:57.440>it's <00:07:57.615>super <00:07:57.789>helpful.

139
00:08:00.010 --> 00:08:03.830
<00:08:00.010>What <00:08:00.117>are <00:08:00.225>other <00:08:00.333>resources <00:08:00.440>that <00:08:00.548>you <00:08:00.656>actually <00:08:00.763>get <00:08:00.871>inspired <00:08:00.979>by, <00:08:01.086>that <00:08:01.194>you <00:08:01.302>actually <00:08:01.409>look

140
00:08:03.870 --> 00:08:06.650
<00:08:03.870>at <00:08:04.045>to <00:08:04.220>learn <00:08:04.395>what <00:08:04.570>you're <00:08:04.745>doing? <00:08:04.920>And <00:08:05.095>is <00:08:05.270>there

141
00:08:07.950 --> 00:08:11.510
<00:08:07.950>something <00:08:08.043>that <00:08:08.136>you <00:08:08.229>would <00:08:08.323>like <00:08:08.416>people <00:08:08.510>to <00:08:08.603>take <00:08:08.696>a <00:08:08.789>look <00:08:08.883>at <00:08:08.976>first <00:08:09.070>before <00:08:09.163>they <00:08:09.256>even <00:08:09.349>start

142
00:08:11.530 --> 00:08:14.030
<00:08:11.530>thinking <00:08:11.705>about <00:08:11.880>putting <00:08:12.054>their <00:08:12.229>first <00:08:12.405>agent <00:08:12.580>in <00:08:12.755>the <00:08:12.929>field?

143
00:08:14.910 --> 00:08:18.630
<00:08:14.910>Yeah, <00:08:15.050>there <00:08:15.190>are <00:08:15.330>a <00:08:15.470>couple <00:08:15.610>of <00:08:15.750>resources <00:08:15.890>which <00:08:16.030>I <00:08:16.170>highly <00:08:16.310>recommend.

144
00:08:18.690 --> 00:08:22.440
<00:08:18.690>Well, <00:08:18.865>the <00:08:19.040>first <00:08:19.215>thing <00:08:19.390>that <00:08:19.565>I <00:08:19.740>would <00:08:19.915>recommend <00:08:20.090>is

145
00:08:22.530 --> 00:08:26.270
<00:08:22.530>start <00:08:22.763>from <00:08:22.996>the <00:08:23.229>basics. <00:08:23.463>So <00:08:23.696>understand <00:08:23.929>the

146
00:08:26.310 --> 00:08:30.270
<00:08:26.310>details <00:08:26.403>of <00:08:26.496>what <00:08:26.590>the <00:08:26.683>system <00:08:26.776>you <00:08:26.870>are <00:08:26.963>working <00:08:27.056>with, <00:08:27.150>and <00:08:27.243>what <00:08:27.336>is <00:08:27.430>the <00:08:27.523>system <00:08:27.616>that <00:08:27.710>you're

147
00:08:30.310 --> 00:08:34.310
<00:08:30.310>building. <00:08:30.450>So <00:08:30.590>understanding <00:08:30.730>what <00:08:30.870>is <00:08:31.010>an <00:08:31.150>LLM, <00:08:31.290>how <00:08:31.430>it <00:08:31.570>works, <00:08:31.710>what

148
00:08:34.390 --> 00:08:38.370
<00:08:34.390>are <00:08:34.590>the <00:08:34.789>security <00:08:34.990>considerations, <00:08:35.189>what <00:08:35.390>are <00:08:35.590>the <00:08:35.789>implications

149
00:08:38.430 --> 00:08:39.691
<00:08:38.429>of <00:08:38.713>using <00:08:38.997>such <00:08:39.281>a <00:08:39.564>system.

150
00:08:40.750 --> 00:08:44.530
<00:08:40.750>And <00:08:40.857>then <00:08:40.965>there <00:08:41.073>is <00:08:41.180>a <00:08:41.288>book <00:08:41.396>which <00:08:41.503>is <00:08:41.611>called, <00:08:41.719>let <00:08:41.826>me <00:08:41.934>see <00:08:42.042>whether <00:08:42.150>I

151
00:08:44.590 --> 00:08:46.950
<00:08:44.590>recall <00:08:44.823>it. <00:08:45.056>It's <00:08:45.290>called <00:08:45.523>"Agentic <00:08:45.756>Design <00:08:45.990>Patterns,"

152
00:08:48.230 --> 00:08:51.370
<00:08:48.230>which <00:08:48.430>is <00:08:48.630>a <00:08:48.830>very <00:08:49.030>interesting <00:08:49.230>book, <00:08:49.430>which <00:08:49.630>explains

153
00:08:51.430 --> 00:08:55.290
<00:08:51.430>how <00:08:51.546>we <00:08:51.663>can <00:08:51.780>build <00:08:51.896>agents <00:08:52.013>and <00:08:52.130>what <00:08:52.246>are <00:08:52.363>some <00:08:52.479>of <00:08:52.596>the <00:08:52.713>patterns <00:08:52.829>of

154
00:08:55.350 --> 00:08:58.690
<00:08:55.350>building <00:08:55.477>those <00:08:55.604>agents, <00:08:55.731>which <00:08:55.859>is <00:08:55.986>also <00:08:56.113>very <00:08:56.240>practical, <00:08:56.368>so <00:08:56.495>I <00:08:56.622>enjoyed <00:08:56.750>it.

155
00:08:59.610 --> 00:09:03.290
<00:08:59.610>And <00:08:59.750>then, <00:08:59.890>again, <00:09:00.030>there <00:09:00.170>are <00:09:00.310>a <00:09:00.450>lot <00:09:00.590>of <00:09:00.730>things <00:09:00.870>on <00:09:01.010>documentation.

156
00:09:03.570 --> 00:09:06.880
<00:09:03.570>So <00:09:03.663>if <00:09:03.756>you <00:09:03.850>are <00:09:03.943>using, <00:09:04.036>for <00:09:04.130>example, <00:09:04.223>in <00:09:04.316>my <00:09:04.410>case, <00:09:04.503>I'm <00:09:04.596>using <00:09:04.690>Master, <00:09:04.783>which <00:09:04.876>is <00:09:04.970>a

157
00:09:06.950 --> 00:09:10.600
<00:09:06.950>TypeScript <00:09:07.230>framework. <00:09:07.510>But <00:09:07.790>these <00:09:08.070>days, <00:09:08.350>every

158
00:09:10.650 --> 00:09:14.190
<00:09:10.650>framework <00:09:10.883>has <00:09:11.116>its <00:09:11.350>own <00:09:11.583>things <00:09:11.816>like <00:09:12.050>tracing,

159
00:09:15.410 --> 00:09:18.000
<00:09:15.410>observability, <00:09:15.689>building <00:09:15.969>evals, <00:09:16.250>building <00:09:16.530>data <00:09:16.810>sets.

160
00:09:18.510 --> 00:09:22.230
<00:09:18.510>So <00:09:18.665>you <00:09:18.821>can <00:09:18.976>use <00:09:19.132>a <00:09:19.287>lot <00:09:19.443>of <00:09:19.598>that. <00:09:19.754>And <00:09:19.910>there

161
00:09:22.290 --> 00:09:26.150
<00:09:22.290>is <00:09:22.406>a <00:09:22.523>book <00:09:22.640>from <00:09:22.756>O'Reilly, <00:09:22.873>which <00:09:22.990>has <00:09:23.106>not <00:09:23.223>been <00:09:23.339>released <00:09:23.456>yet, <00:09:23.573>but <00:09:23.689>I'm

162
00:09:26.190 --> 00:09:29.790
<00:09:26.190>following <00:09:26.290>the <00:09:26.390>author, <00:09:26.490>which <00:09:26.590>I <00:09:26.690>cannot <00:09:26.790>recall <00:09:26.890>the <00:09:26.990>name <00:09:27.090>at <00:09:27.190>this <00:09:27.290>moment, <00:09:27.390>which <00:09:27.490>is <00:09:27.590>about

163
00:09:29.830 --> 00:09:33.520
<00:09:29.830>building <00:09:30.005>evals. <00:09:30.180>Because <00:09:30.355>building <00:09:30.530>evals <00:09:30.705>and <00:09:30.880>golden <00:09:31.055>data <00:09:31.230>sets,

164
00:09:34.750 --> 00:09:38.150
<00:09:34.750>it's <00:09:34.877>way <00:09:35.004>harder <00:09:35.131>than <00:09:35.259>it <00:09:35.386>looks <00:09:35.513>like, <00:09:35.640>and <00:09:35.768>it's <00:09:35.895>a <00:09:36.022>continuous <00:09:36.150>effort.

165
00:09:38.810 --> 00:09:38.820
Mm.

166
00:09:38.870 --> 00:09:42.610
<00:09:38.870>So <00:09:39.025>ensuring <00:09:39.181>that <00:09:39.336>your <00:09:39.492>system <00:09:39.647>is <00:09:39.803>testable, <00:09:39.958>it's <00:09:40.114>something <00:09:40.270>super

167
00:09:42.650 --> 00:09:46.619
<00:09:42.650>important, <00:09:42.777>and <00:09:42.904>people <00:09:43.031>usually <00:09:43.159>skip <00:09:43.286>it <00:09:43.413>because <00:09:43.540>they <00:09:43.668>try <00:09:43.795>it <00:09:43.922>once <00:09:44.050>with

168
00:09:46.630 --> 00:09:50.290
<00:09:46.630>their <00:09:46.805>chat-ish <00:09:46.980>interface, <00:09:47.155>and <00:09:47.330>they <00:09:47.505>say, <00:09:47.680>"Yeah, <00:09:47.855>this <00:09:48.030>works."

169
00:09:50.790 --> 00:09:53.450
<00:09:50.790>But <00:09:50.930>then <00:09:51.069>they <00:09:51.209>don't <00:09:51.349>have <00:09:51.490>an <00:09:51.630>automated <00:09:51.770>way <00:09:51.910>to <00:09:52.050>test <00:09:52.189>it.

170
00:09:55.610 --> 00:09:59.270
<00:09:55.610>There <00:09:55.750>is, <00:09:55.890>of <00:09:56.030>course, <00:09:56.170>the <00:09:56.310>big <00:09:56.450>story <00:09:56.590>about <00:09:56.730>AI <00:09:56.870>making <00:09:57.010>it

171
00:10:00.530 --> 00:10:04.330
<00:10:00.530>easy <00:10:00.637>for <00:10:00.745>anybody <00:10:00.853>to <00:10:00.960>start <00:10:01.068>building <00:10:01.176>software <00:10:01.283>and <00:10:01.391>to <00:10:01.499>start <00:10:01.606>building <00:10:01.714>solutions <00:10:01.822>and <00:10:01.930>not

172
00:10:04.390 --> 00:10:05.870
<00:10:04.390>even <00:10:04.723>thinking <00:10:05.056>about <00:10:05.389>the <00:10:05.722>software.

173
00:10:06.420 --> 00:10:10.390
<00:10:06.420>The <00:10:06.547>founder <00:10:06.674>of <00:10:06.801>Kubernetes <00:10:06.929>just <00:10:07.056>said <00:10:07.183>that <00:10:07.310>AI-generated <00:10:07.438>code <00:10:07.565>is <00:10:07.692>going <00:10:07.819>to

174
00:10:10.430 --> 00:10:12.649
<00:10:10.430>be <00:10:10.663>invisible, <00:10:10.896>much <00:10:11.130>like <00:10:11.363>assembly <00:10:11.596>code <00:10:11.829>is.

175
00:10:13.410 --> 00:10:16.550
<00:10:13.410>People <00:10:13.526>never <00:10:13.643>look <00:10:13.760>at <00:10:13.876>the <00:10:13.993>code <00:10:14.110>anymore, <00:10:14.226>it's <00:10:14.343>just <00:10:14.459>got <00:10:14.576>to <00:10:14.693>be <00:10:14.810>executed.

176
00:10:17.150 --> 00:10:21.040
<00:10:17.150>So <00:10:17.324>this <00:10:17.500>narrative <00:10:17.675>is <00:10:17.850>in <00:10:18.025>the <00:10:18.199>headlines <00:10:18.375>everywhere, <00:10:18.550>that

177
00:10:21.070 --> 00:10:24.970
<00:10:21.070>anybody <00:10:21.210>can <00:10:21.350>become <00:10:21.490>a <00:10:21.630>software <00:10:21.770>developer, <00:10:21.910>anybody <00:10:22.050>can <00:10:22.190>become <00:10:22.330>a <00:10:22.470>product

178
00:10:25.010 --> 00:10:28.530
<00:10:25.010>creator, <00:10:25.150>that <00:10:25.290>basically <00:10:25.430>let <00:10:25.569>an <00:10:25.710>agent <00:10:25.850>do <00:10:25.990>the <00:10:26.130>work <00:10:26.270>for <00:10:26.410>you,

179
00:10:28.970 --> 00:10:31.570
<00:10:28.970>use <00:10:29.145>some <00:10:29.320>framework <00:10:29.495>that <00:10:29.670>already <00:10:29.845>gives <00:10:30.020>you <00:10:30.195>an <00:10:30.370>agent.

180
00:10:32.070 --> 00:10:34.450
<00:10:32.070>Now <00:10:32.225>you're <00:10:32.381>telling <00:10:32.536>us <00:10:32.692>it's <00:10:32.847>not <00:10:33.003>as <00:10:33.158>easy <00:10:33.314>as <00:10:33.470>that.

181
00:10:34.490 --> 00:10:38.430
<00:10:34.490>Do <00:10:34.665>you <00:10:34.840>think <00:10:35.015>that <00:10:35.190>we <00:10:35.365>need <00:10:35.540>a <00:10:35.715>moment <00:10:35.890>where

182
00:10:38.450 --> 00:10:42.370
<00:10:38.450>people <00:10:38.590>realize <00:10:38.730>that, <00:10:38.870>okay, <00:10:39.010>just <00:10:39.150>getting <00:10:39.290>it <00:10:39.430>off <00:10:39.570>the <00:10:39.710>shelf <00:10:39.850>is

183
00:10:42.390 --> 00:10:46.130
<00:10:42.390>probably <00:10:42.506>not <00:10:42.623>the <00:10:42.740>best <00:10:42.856>idea, <00:10:42.973>but <00:10:43.090>building <00:10:43.206>the <00:10:43.323>thing <00:10:43.439>and <00:10:43.556>understanding <00:10:43.673>what <00:10:43.790>you're

184
00:10:46.150 --> 00:10:47.770
<00:10:46.150>building <00:10:46.383>is <00:10:46.616>still <00:10:46.850>a <00:10:47.083>skill <00:10:47.316>that's <00:10:47.550>needed?

185
00:10:48.790 --> 00:10:52.330
<00:10:48.790>Yeah. <00:10:48.930>Of <00:10:49.069>course, <00:10:49.209>I <00:10:49.349>cannot <00:10:49.490>predict <00:10:49.630>the <00:10:49.770>future <00:10:49.910>, <00:10:50.050>but <00:10:50.189>at

186
00:10:52.390 --> 00:10:54.050
<00:10:52.390>least <00:10:53.790>today,

187
00:10:54.970 --> 00:10:58.270
<00:10:54.970>I <00:10:55.145>completely <00:10:55.320>agree <00:10:55.495>that <00:10:55.670>at <00:10:55.845>least <00:10:56.020>part <00:10:56.195>of <00:10:56.370>the

188
00:10:59.130 --> 00:10:59.590
<00:10:59.130>code <00:10:59.337>that <00:10:59.544>we

189
00:11:00.710 --> 00:11:03.490
<00:11:00.710>are <00:11:00.865>writing, <00:11:01.021>we <00:11:01.176>don't <00:11:01.332>have <00:11:01.487>to <00:11:01.643>review <00:11:01.798>everything <00:11:01.954>these <00:11:02.110>days.

190
00:11:03.790 --> 00:11:05.970
<00:11:03.790>At <00:11:04.069>least <00:11:04.349>I'm <00:11:04.630>not <00:11:04.910>reviewing <00:11:05.189>everything.

191
00:11:06.250 --> 00:11:10.210
<00:11:06.250>But <00:11:06.377>you <00:11:06.504>have <00:11:06.631>a <00:11:06.759>set <00:11:06.886>of <00:11:07.013>guardrails. <00:11:07.140>And <00:11:07.268>of <00:11:07.395>course, <00:11:07.522>the <00:11:07.650>code

192
00:11:10.230 --> 00:11:13.880
<00:11:10.230>that <00:11:10.370>you <00:11:10.510>are <00:11:10.650>not <00:11:10.790>reviewing <00:11:10.930>maybe <00:11:11.070>is <00:11:11.210>used <00:11:11.350>for <00:11:11.490>dim <00:11:11.630>applications,

193
00:11:13.990 --> 00:11:17.310
<00:11:13.990>POCs, <00:11:14.145>but <00:11:14.301>it's <00:11:14.456>not <00:11:14.612>something <00:11:14.767>that <00:11:14.923>we <00:11:15.078>deploy <00:11:15.234>in <00:11:15.390>production,

194
00:11:17.810 --> 00:11:18.170
hopefully.

195
00:11:19.990 --> 00:11:23.250
<00:11:19.990>So <00:11:20.106>yeah, <00:11:20.223>I <00:11:20.340>think <00:11:20.456>there <00:11:20.573>is <00:11:20.690>still <00:11:20.806>a <00:11:20.923>gap, <00:11:21.040>and <00:11:21.156>I <00:11:21.273>think <00:11:21.390>people--

196
00:11:23.930 --> 00:11:25.250
<00:11:23.930>I'm <00:11:24.326>not <00:11:24.722>sure <00:11:25.117>I

197
00:11:26.130 --> 00:11:30.050
<00:11:26.130>like <00:11:26.285>this <00:11:26.441>thing, <00:11:26.596>at <00:11:26.752>some <00:11:26.907>point <00:11:27.063>we <00:11:27.218>are <00:11:27.374>going <00:11:27.530>to

198
00:11:30.130 --> 00:11:32.290
<00:11:30.130>see <00:11:30.330>codes <00:11:30.530>as <00:11:30.730>we <00:11:30.930>are <00:11:31.130>now <00:11:31.330>seeing <00:11:31.530>assembly.

199
00:11:32.530 --> 00:11:36.140
<00:11:32.530>Because <00:11:32.704>the <00:11:32.880>difference <00:11:33.055>between <00:11:33.230>assembly <00:11:33.405>and <00:11:33.579>C <00:11:33.755>code <00:11:33.930>and

200
00:11:36.170 --> 00:11:39.930
<00:11:36.170>maybe <00:11:36.297>Python <00:11:36.424>code <00:11:36.551>on <00:11:36.679>top <00:11:36.806>or <00:11:36.933>whatever, <00:11:37.060>is <00:11:37.188>that <00:11:37.315>they <00:11:37.442>are <00:11:37.569>still

201
00:11:39.990 --> 00:11:43.130
<00:11:39.990>deterministic. <00:11:40.223>So <00:11:40.456>I <00:11:40.690>compile <00:11:40.923>my <00:11:41.156>C <00:11:41.390>code

202
00:11:43.450 --> 00:11:45.350
<00:11:43.450>deterministic <00:11:43.916>way <00:11:44.383>into <00:11:44.850>assembly.

203
00:11:46.670 --> 00:11:48.410
<00:11:46.670>But <00:11:46.903>down <00:11:47.136>here, <00:11:47.370>we <00:11:47.603>do <00:11:47.836>have <00:11:48.069>this

204
00:11:50.130 --> 00:11:52.010
<00:11:50.130>conversion, <00:11:50.480>which <00:11:50.830>is <00:11:51.180>not <00:11:51.530>deterministic.

205
00:11:52.150 --> 00:11:55.730
<00:11:52.150>So <00:11:52.277>I <00:11:52.404>can <00:11:52.531>say <00:11:52.659>to <00:11:52.786>my <00:11:52.913>agent, <00:11:53.040>"Hey, <00:11:53.168>build <00:11:53.295>a <00:11:53.422>dashboard." <00:11:53.550>Lovely.

206
00:11:56.410 --> 00:12:00.250
<00:11:56.410>It <00:11:56.550>can <00:11:56.689>definitely <00:11:56.829>escape <00:11:56.969>authentication, <00:11:57.110>or <00:11:57.250>it <00:11:57.390>can <00:11:57.530>expose <00:11:57.670>in <00:11:57.810>the

207
00:12:00.270 --> 00:12:02.010
<00:12:00.270>front-end <00:12:00.550>something <00:12:00.829>that <00:12:01.110>shouldn't <00:12:01.390>be <00:12:01.670>exposed.

208
00:12:02.890 --> 00:12:04.490
<00:12:02.890>So <00:12:03.123>who <00:12:03.356>is <00:12:03.590>in <00:12:03.823>charge <00:12:04.056>of <00:12:04.290>that?

209
00:12:05.450 --> 00:12:08.690
<00:12:05.450>That's <00:12:05.550>the <00:12:05.650>human <00:12:05.750>in <00:12:05.850>the <00:12:05.950>loop. <00:12:06.050>The <00:12:06.150>human <00:12:06.250>in <00:12:06.350>the <00:12:06.450>loop <00:12:06.550>is <00:12:06.650>seen <00:12:06.750>as <00:12:06.850>a

210
00:12:08.770 --> 00:12:11.770
<00:12:08.770>deterrent <00:12:09.120>of <00:12:09.470>making <00:12:09.819>things <00:12:10.170>slower.

211
00:12:12.330 --> 00:12:16.150
<00:12:12.330>But <00:12:12.446>I <00:12:12.563>think <00:12:12.680>the <00:12:12.796>skill <00:12:12.913>set <00:12:13.030>that <00:12:13.146>we <00:12:13.263>need <00:12:13.380>nowadays <00:12:13.496>more <00:12:13.613>is <00:12:13.730>actually

212
00:12:16.250 --> 00:12:20.130
<00:12:16.250>be <00:12:16.377>an <00:12:16.504>observer <00:12:16.631>and <00:12:16.759>be <00:12:16.886>somebody <00:12:17.013>who <00:12:17.140>actually <00:12:17.268>reviews <00:12:17.395>the <00:12:17.522>stuff <00:12:17.650>that

213
00:12:20.170 --> 00:12:22.200
<00:12:20.170>gets <00:12:20.370>generated <00:12:20.569>or <00:12:20.770>what <00:12:20.969>the <00:12:21.170>agents <00:12:21.370>are <00:12:21.569>doing.

214
00:12:22.630 --> 00:12:26.280
<00:12:22.630>So <00:12:22.730>it <00:12:22.830>feels <00:12:22.930>like <00:12:23.030>being <00:12:23.130>a <00:12:23.230>lead <00:12:23.330>developer <00:12:23.430>for <00:12:23.530>your <00:12:23.630>own <00:12:23.730>agents <00:12:23.830>all <00:12:23.930>the <00:12:24.030>time,

215
00:12:26.870 --> 00:12:28.830
<00:12:26.870>and <00:12:27.010>that <00:12:27.150>means <00:12:27.290>you <00:12:27.430>need <00:12:27.570>a <00:12:27.710>skill <00:12:27.850>set <00:12:27.990>of <00:12:28.130>a <00:12:28.270>developer.

216
00:12:29.542 --> 00:12:32.002
<00:12:29.542>Yeah, <00:12:29.822>pretty <00:12:30.102>much. <00:12:30.382>I <00:12:30.662>think <00:12:30.942>that

217
00:12:34.742 --> 00:12:38.182
<00:12:34.742>what's <00:12:34.849>going <00:12:34.957>to <00:12:35.065>happen <00:12:35.172>in <00:12:35.280>the <00:12:35.388>future, <00:12:35.495>or <00:12:35.603>what's <00:12:35.711>happening <00:12:35.818>right <00:12:35.926>now <00:12:36.034>with <00:12:36.141>teams

218
00:12:38.222 --> 00:12:42.182
<00:12:38.222>which <00:12:38.362>are <00:12:38.502>using <00:12:38.641>AI <00:12:38.781>and <00:12:38.922>which <00:12:39.062>are <00:12:39.202>maybe <00:12:39.342>starting <00:12:39.482>to <00:12:39.622>use

219
00:12:42.682 --> 00:12:46.222
<00:12:42.682>multiple <00:12:42.837>agents <00:12:42.993>in <00:12:43.148>parallel, <00:12:43.304>which <00:12:43.459>I <00:12:43.615>think <00:12:43.770>we <00:12:43.926>cannot <00:12:44.082>parallelize

220
00:12:46.542 --> 00:12:50.342
<00:12:46.542>too <00:12:46.642>much, <00:12:46.742>to <00:12:46.842>be <00:12:46.942>fair, <00:12:47.042>because <00:12:47.142>there <00:12:47.242>is <00:12:47.342>a <00:12:47.442>cognitive <00:12:47.542>load <00:12:47.642>in <00:12:47.742>managing <00:12:47.842>a <00:12:47.942>single

221
00:12:50.382 --> 00:12:52.122
<00:12:50.382>agent <00:12:50.848>and <00:12:51.315>multiple <00:12:51.781>agents.

222
00:12:53.322 --> 00:12:57.302
<00:12:53.322>I <00:12:53.429>think <00:12:53.537>that <00:12:53.645>every <00:12:53.752>developer <00:12:53.860>is <00:12:53.968>going <00:12:54.075>to <00:12:54.183>become <00:12:54.291>more <00:12:54.398>like <00:12:54.506>a <00:12:54.614>tech <00:12:54.722>lead

223
00:12:57.382 --> 00:13:01.052
<00:12:57.382>of <00:12:57.521>their <00:12:57.661>own <00:12:57.801>small <00:12:57.941>agents <00:12:58.082>team, <00:12:58.222>and <00:12:58.362>then <00:12:58.502>they <00:12:58.641>do <00:12:58.781>the

224
00:13:01.122 --> 00:13:04.682
<00:13:01.122>review, <00:13:01.322>they <00:13:01.521>give <00:13:01.722>the <00:13:01.921>direction, <00:13:02.122>they <00:13:02.322>do <00:13:02.521>all

225
00:13:04.742 --> 00:13:08.082
<00:13:04.742>these <00:13:04.975>things <00:13:05.208>which <00:13:05.442>tech <00:13:05.675>leads <00:13:05.908>did <00:13:06.141>for

226
00:13:08.122 --> 00:13:09.502
<00:13:08.122>decades <00:13:09.363>now.

227
00:13:10.312 --> 00:13:14.102
<00:13:10.312>But <00:13:10.452>then <00:13:10.592>every <00:13:10.732>developer <00:13:10.872>can <00:13:11.012>be <00:13:11.152>the <00:13:11.292>small <00:13:11.432>tech <00:13:11.572>lead <00:13:11.712>of

228
00:13:14.162 --> 00:13:17.862
<00:13:14.162>their <00:13:14.289>team <00:13:14.416>of <00:13:14.543>agents. <00:13:14.671>And <00:13:14.798>we <00:13:14.925>are <00:13:15.052>seeing <00:13:15.180>that <00:13:15.307>working <00:13:15.434>well <00:13:15.562>across

229
00:13:17.922 --> 00:13:21.842
<00:13:17.922>multiple <00:13:18.038>teams <00:13:18.155>right <00:13:18.272>now <00:13:18.388>at <00:13:18.505>NearForm, <00:13:18.622>and <00:13:18.738>we <00:13:18.855>are <00:13:18.972>trying <00:13:19.088>to <00:13:19.205>expand <00:13:19.322>the

230
00:13:21.882 --> 00:13:24.882
<00:13:21.882>usage <00:13:22.082>of <00:13:22.281>AI <00:13:22.482>across <00:13:22.681>the <00:13:22.882>entire <00:13:23.082>SDLC, <00:13:23.281>actually.

231
00:13:27.002 --> 00:13:30.962
<00:13:27.002>Now, <00:13:27.141>one <00:13:27.281>of <00:13:27.421>the <00:13:27.561>things <00:13:27.702>that <00:13:27.842>got <00:13:27.982>this <00:13:28.122>whole <00:13:28.262>thing <00:13:28.401>started

232
00:13:31.022 --> 00:13:34.821
<00:13:31.022>was <00:13:31.177>basically <00:13:31.333>OpenClau <00:13:31.488>and <00:13:31.644>actually <00:13:31.799>using <00:13:31.955>your <00:13:32.110>own <00:13:32.266>machines <00:13:32.422>to

233
00:13:34.862 --> 00:13:38.802
<00:13:34.862>run <00:13:34.969>these <00:13:35.077>kind <00:13:35.185>of <00:13:35.292>environments. <00:13:35.400>I <00:13:35.508>find <00:13:35.615>it <00:13:35.723>interesting <00:13:35.831>that <00:13:35.938>people <00:13:36.046>are <00:13:36.154>just <00:13:36.262>very

234
00:13:38.862 --> 00:13:42.672
<00:13:38.862>happily <00:13:38.978>giving <00:13:39.095>access <00:13:39.212>to <00:13:39.328>everything <00:13:39.445>on <00:13:39.562>their <00:13:39.678>machine <00:13:39.795>to <00:13:39.911>some <00:13:40.028>piece <00:13:40.145>of <00:13:40.262>software,

235
00:13:43.622 --> 00:13:46.102
<00:13:43.622>while <00:13:43.796>they're <00:13:43.972>actually <00:13:44.146>wondering <00:13:44.322>where <00:13:44.497>to <00:13:44.671>download <00:13:44.847>a <00:13:45.021>browser.

236
00:13:47.942 --> 00:13:51.912
<00:13:47.942>But <00:13:48.097>is <00:13:48.253>the <00:13:48.408>danger <00:13:48.564>right <00:13:48.719>now <00:13:48.875>that <00:13:49.030>we're <00:13:49.186>actually <00:13:49.342>putting

237
00:13:51.962 --> 00:13:55.502
<00:13:51.962>so <00:13:52.117>many <00:13:52.273>abstractions <00:13:52.428>on <00:13:52.584>top <00:13:52.739>that <00:13:52.895>basically <00:13:53.050>seems <00:13:53.206>a <00:13:53.362>human

238
00:13:55.542 --> 00:13:59.402
<00:13:55.542>person, <00:13:55.658>which <00:13:55.775>is <00:13:55.892>not <00:13:56.008>a <00:13:56.125>human <00:13:56.242>person, <00:13:56.358>just <00:13:56.475>an <00:13:56.592>LLM <00:13:56.708>giving <00:13:56.825>us <00:13:56.942>answers,

239
00:13:59.442 --> 00:14:03.362
<00:13:59.442>so <00:13:59.569>people <00:13:59.696>forget <00:13:59.823>that <00:13:59.951>the <00:14:00.078>access <00:14:00.205>to <00:14:00.332>their <00:14:00.460>machine <00:14:00.587>should <00:14:00.714>be <00:14:00.842>restricted

240
00:14:03.422 --> 00:14:05.622
<00:14:03.422>to <00:14:03.655>only <00:14:03.888>things <00:14:04.122>that <00:14:04.355>really <00:14:04.588>make <00:14:04.822>sense?

241
00:14:06.802 --> 00:14:07.742
<00:14:06.802>Yeah, <00:14:07.225>and <00:14:07.647>that's

242
00:14:08.802 --> 00:14:10.402
<00:14:08.802>exactly <00:14:09.268>the <00:14:09.735>problem. <00:14:10.202>Of

243
00:14:12.642 --> 00:14:16.602
<00:14:12.642>course, <00:14:12.769>I'm <00:14:12.896>exaggerating <00:14:13.023>here, <00:14:13.151>but <00:14:13.278>you <00:14:13.405>have <00:14:13.532>pretty <00:14:13.660>much <00:14:13.787>the <00:14:13.914>same <00:14:14.042>when

244
00:14:16.622 --> 00:14:19.642
<00:14:16.622>you <00:14:16.822>have <00:14:17.021>bypass <00:14:17.222>permissions <00:14:17.421>on <00:14:17.622>Cloud <00:14:17.822>Code <00:14:18.021>or

245
00:14:19.762 --> 00:14:23.182
<00:14:19.762>Yolo <00:14:20.041>mode <00:14:20.321>in <00:14:20.602>other <00:14:20.882>tools, <00:14:21.161>where

246
00:14:23.922 --> 00:14:27.662
<00:14:23.922>basically <00:14:24.122>the <00:14:24.322>moment <00:14:24.522>you <00:14:24.722>give <00:14:24.922>access <00:14:25.122>to <00:14:25.322>an

247
00:14:27.802 --> 00:14:30.482
<00:14:27.802>LLM, <00:14:28.152>to <00:14:28.502>your <00:14:28.852>entire <00:14:29.202>system,

248
00:14:32.682 --> 00:14:35.562
<00:14:32.682>you <00:14:32.857>might <00:14:33.032>have <00:14:33.207>a <00:14:33.382>lot <00:14:33.557>of <00:14:33.732>security <00:14:33.907>issues, <00:14:34.082>right?

249
00:14:35.662 --> 00:14:38.722
<00:14:35.662>Because <00:14:35.789>you <00:14:35.916>are <00:14:36.043>giving <00:14:36.171>access <00:14:36.298>to <00:14:36.425>your, <00:14:36.552>I <00:14:36.680>don't <00:14:36.807>know, <00:14:36.934>SH <00:14:37.062>keys,

250
00:14:40.082 --> 00:14:41.832
<00:14:40.082>whatever <00:14:40.362>on <00:14:40.641>your <00:14:40.922>file <00:14:41.202>system, <00:14:41.482>right?

251
00:14:42.482 --> 00:14:46.302
<00:14:42.482>So <00:14:42.715>I <00:14:42.948>love <00:14:43.182>these <00:14:43.415>things <00:14:43.648>for, <00:14:43.882>let's

252
00:14:46.342 --> 00:14:49.902
<00:14:46.342>say, <00:14:46.575>experiments <00:14:46.808>and <00:14:47.042>testing. <00:14:47.275>They <00:14:47.508>are <00:14:47.742>great.

253
00:14:50.702 --> 00:14:54.582
<00:14:50.702>I <00:14:50.795>can <00:14:50.888>see <00:14:50.982>that <00:14:51.075>at <00:14:51.168>some <00:14:51.262>point <00:14:51.355>in <00:14:51.448>the <00:14:51.542>future, <00:14:51.635>we <00:14:51.728>might <00:14:51.822>have <00:14:51.915>something <00:14:52.008>like <00:14:52.102>this

254
00:14:54.722 --> 00:14:57.902
<00:14:54.722>production-ready, <00:14:55.072>secure, <00:14:55.422>and <00:14:55.771>which <00:14:56.122>we

255
00:14:59.382 --> 00:15:03.282
<00:14:59.382>use <00:14:59.537>in <00:14:59.693>our <00:14:59.848>everyday <00:15:00.004>life. <00:15:00.159>I'm <00:15:00.315>not <00:15:00.470>sure <00:15:00.626>we <00:15:00.781>are

256
00:15:03.302 --> 00:15:06.622
<00:15:03.302>there <00:15:03.442>yet, <00:15:03.582>at <00:15:03.722>least <00:15:03.862>in <00:15:04.002>terms <00:15:04.142>of <00:15:04.282>maturity <00:15:04.422>of <00:15:04.562>the <00:15:04.702>ecosystem.

257
00:15:06.702 --> 00:15:10.571
<00:15:06.702>So <00:15:06.829>at <00:15:06.956>the <00:15:07.083>moment, <00:15:07.211>the <00:15:07.338>recommendation <00:15:07.465>that <00:15:07.592>I <00:15:07.720>would <00:15:07.847>give <00:15:07.974>is <00:15:08.102>let's

258
00:15:10.622 --> 00:15:14.562
<00:15:10.622>experiment <00:15:10.729>with <00:15:10.837>OpenClau <00:15:10.945>and <00:15:11.052>these <00:15:11.160>kind <00:15:11.268>of <00:15:11.375>tools <00:15:11.483>because <00:15:11.591>we <00:15:11.698>don't <00:15:11.806>want <00:15:11.914>to <00:15:12.021>get

259
00:15:14.602 --> 00:15:17.282
<00:15:14.602>left <00:15:14.742>behind, <00:15:14.882>and <00:15:15.021>we <00:15:15.161>want <00:15:15.302>to <00:15:15.442>see <00:15:15.582>how <00:15:15.722>these <00:15:15.862>things <00:15:16.002>work.

260
00:15:17.962 --> 00:15:21.742
<00:15:17.962>But <00:15:18.069>just <00:15:18.177>don't <00:15:18.285>do <00:15:18.392>it <00:15:18.500>on <00:15:18.608>your <00:15:18.715>personal <00:15:18.823>machine <00:15:18.931>where <00:15:19.038>you <00:15:19.146>have <00:15:19.254>your <00:15:19.362>keys

261
00:15:21.802 --> 00:15:22.682
<00:15:21.802>and <00:15:22.066>stuff <00:15:22.330>like <00:15:22.594>this.

262
00:15:23.922 --> 00:15:27.182
<00:15:23.922>Well, <00:15:24.077>there <00:15:24.233>are <00:15:24.388>systems <00:15:24.544>like <00:15:24.699>Nemo <00:15:24.855>Clau <00:15:25.010>from <00:15:25.166>NVIDIA <00:15:25.322>or

263
00:15:27.242 --> 00:15:31.022
<00:15:27.242>NoNo.sh. <00:15:27.416>We <00:15:27.592>had <00:15:27.766>that <00:15:27.942>person <00:15:28.117>on <00:15:28.291>the <00:15:28.467>show <00:15:28.641>last

264
00:15:31.082 --> 00:15:34.762
<00:15:31.082>week. <00:15:31.198>I <00:15:31.315>think <00:15:31.432>there <00:15:31.548>are <00:15:31.665>guardrail <00:15:31.782>systems <00:15:31.898>that <00:15:32.015>actually <00:15:32.132>say, <00:15:32.248>"Oh, <00:15:32.365>make <00:15:32.482>a

265
00:15:34.842 --> 00:15:38.682
<00:15:34.842>container, <00:15:34.969>and <00:15:35.096>let <00:15:35.223>it <00:15:35.351>only <00:15:35.478>stay <00:15:35.605>in <00:15:35.732>there." <00:15:35.860>Then <00:15:35.987>again, <00:15:36.114>I've <00:15:36.242>heard

266
00:15:38.722 --> 00:15:42.351
<00:15:38.722>horror <00:15:38.877>stories <00:15:39.033>that <00:15:39.188>basically <00:15:39.344>Clau <00:15:39.499>is <00:15:39.655>really <00:15:39.810>hard <00:15:39.966>to <00:15:40.122>stop

267
00:15:43.702 --> 00:15:46.182
<00:15:43.702>from <00:15:44.052>indexing <00:15:44.402>your <00:15:44.752>environment <00:15:45.102>files.

268
00:15:46.242 --> 00:15:49.702
<00:15:46.242>Yeah. <00:15:46.397>Again, <00:15:46.553>at <00:15:46.708>the <00:15:46.864>moment, <00:15:47.019>we <00:15:47.175>are <00:15:47.330>in <00:15:47.486>this <00:15:47.641>magical

269
00:15:49.722 --> 00:15:52.602
<00:15:49.722>moment <00:15:49.955>where <00:15:50.188>there <00:15:50.422>is <00:15:50.655>no <00:15:50.888>silver <00:15:51.122>bullet.

270
00:15:53.052 --> 00:15:57.042
<00:15:53.052>And <00:15:53.207>so <00:15:53.363>even <00:15:53.518>the <00:15:53.674>more <00:15:53.829>secure <00:15:53.985>solutions, <00:15:54.140>the <00:15:54.296>more <00:15:54.452>brilliant

271
00:15:57.102 --> 00:16:00.642
<00:15:57.102>solutions, <00:15:57.257>there <00:15:57.413>is <00:15:57.568>still <00:15:57.724>one <00:15:57.879>problem, <00:15:58.035>which <00:15:58.190>is <00:15:58.346>we <00:15:58.502>are

272
00:16:00.662 --> 00:16:04.602
<00:16:00.662>relying <00:16:00.895>on <00:16:01.128>non-deterministic <00:16:01.362>systems, <00:16:01.595>which <00:16:01.828>by <00:16:02.062>nature

273
00:16:05.002 --> 00:16:08.442
<00:16:05.002>can <00:16:05.176>be <00:16:05.352>tricked <00:16:05.526>and <00:16:05.702>can <00:16:05.877>do <00:16:06.051>unreliable <00:16:06.227>things. <00:16:06.401>Right?

274
00:16:08.542 --> 00:16:12.342
<00:16:08.542>And <00:16:08.669>that's <00:16:08.796>not <00:16:08.923>a <00:16:09.051>problem <00:16:09.178>of <00:16:09.305>the <00:16:09.432>LLM <00:16:09.560>architecture, <00:16:09.687>it's <00:16:09.814>a <00:16:09.942>characteristic.

275
00:16:12.512 --> 00:16:16.502
<00:16:12.512>Right? <00:16:12.667>It's <00:16:12.823>what <00:16:12.978>makes <00:16:13.134>it <00:16:13.289>special <00:16:13.445>and <00:16:13.600>to <00:16:13.756>some <00:16:13.911>extent

276
00:16:16.562 --> 00:16:17.942
<00:16:16.562>dangerous <00:16:16.872>on <00:16:17.183>the <00:16:17.493>other <00:16:17.804>side.

277
00:16:19.142 --> 00:16:23.002
<00:16:19.142>So <00:16:19.282>yeah. <00:16:19.422>I <00:16:19.562>guess <00:16:19.702>in <00:16:19.842>the <00:16:19.982>future, <00:16:20.122>we <00:16:20.262>will <00:16:20.402>find <00:16:20.542>ways

278
00:16:23.042 --> 00:16:24.062
<00:16:23.042>to, <00:16:23.501>let's <00:16:23.960>say,

279
00:16:25.362 --> 00:16:29.342
<00:16:25.362>have <00:16:25.478>a <00:16:25.595>more <00:16:25.712>mature <00:16:25.828>ecosystem. <00:16:25.945>But <00:16:26.062>I <00:16:26.178>still <00:16:26.295>think <00:16:26.411>that <00:16:26.528>we <00:16:26.645>are <00:16:26.762>not

280
00:16:29.382 --> 00:16:29.782
<00:16:29.382>there <00:16:29.742>yet.

281
00:16:32.182 --> 00:16:35.192
<00:16:32.182>One <00:16:32.298>thing <00:16:32.415>we <00:16:32.532>had <00:16:32.648>when <00:16:32.765>we'd <00:16:32.882>started <00:16:32.998>with <00:16:33.115>chatbots <00:16:33.232>and <00:16:33.348>we <00:16:33.465>started <00:16:33.582>with

282
00:16:35.822 --> 00:16:39.402
<00:16:35.822>ChatGPT <00:16:36.102>is <00:16:36.382>two <00:16:36.662>things <00:16:36.942>came <00:16:37.222>to

283
00:16:39.442 --> 00:16:43.182
<00:16:39.442>pass <00:16:39.582>really <00:16:39.722>quickly. <00:16:39.862>One <00:16:40.002>was <00:16:40.142>that <00:16:40.282>actually <00:16:40.422>hallucinations <00:16:40.562>are <00:16:40.702>a <00:16:40.842>weird

284
00:16:43.302 --> 00:16:46.821
<00:16:43.302>thing, <00:16:43.395>that <00:16:43.488>basically <00:16:43.582>it <00:16:43.675>just <00:16:43.768>tries <00:16:43.862>to <00:16:43.955>give <00:16:44.048>you <00:16:44.142>an <00:16:44.235>answer <00:16:44.328>when <00:16:44.422>it <00:16:44.515>doesn't <00:16:44.608>have <00:16:44.702>an

285
00:16:46.902 --> 00:16:48.802
<00:16:46.902>answer. <00:16:47.182>And <00:16:47.462>the <00:16:47.742>second <00:16:48.022>was <00:16:48.302>the

286
00:16:49.742 --> 00:16:53.522
<00:16:49.742>sycophantic <00:16:49.849>way <00:16:49.957>of <00:16:50.065>doing <00:16:50.172>it, <00:16:50.280>that <00:16:50.388>it's <00:16:50.495>always <00:16:50.603>like, <00:16:50.711>"Oh, <00:16:50.818>that's <00:16:50.926>a <00:16:51.034>great <00:16:51.141>idea.

287
00:16:53.602 --> 00:16:56.782
<00:16:53.602>Thank <00:16:53.742>you <00:16:53.882>so <00:16:54.021>much," <00:16:54.161>and, <00:16:54.302>"Wonderful," <00:16:54.442>and <00:16:54.582>these <00:16:54.722>kind <00:16:54.862>of <00:16:55.002>things.

288
00:16:57.202 --> 00:17:01.102
<00:16:57.202>In <00:16:57.309>a <00:16:57.417>chat <00:16:57.525>system, <00:16:57.632>that's <00:16:57.740>weird <00:16:57.848>enough, <00:16:57.955>but <00:16:58.063>I <00:16:58.171>think <00:16:58.278>in <00:16:58.386>an <00:16:58.494>agent, <00:16:58.602>when

289
00:17:01.122 --> 00:17:04.782
<00:17:01.122>it <00:17:01.296>does <00:17:01.472>that <00:17:01.646>under <00:17:01.822>the <00:17:01.997>hood <00:17:02.171>to <00:17:02.347>randomly <00:17:02.521>hallucinate

290
00:17:04.843 --> 00:17:08.822
<00:17:04.843>access <00:17:04.943>to <00:17:05.043>tools <00:17:05.143>that <00:17:05.243>don't <00:17:05.343>exist <00:17:05.443>or <00:17:05.543>tools <00:17:05.643>that <00:17:05.743>might <00:17:05.843>not <00:17:05.943>be <00:17:06.043>the <00:17:06.143>right <00:17:06.243>thing,

291
00:17:09.222 --> 00:17:10.722
<00:17:09.222>that's <00:17:09.446>the <00:17:09.672>big <00:17:09.896>danger <00:17:10.122>there <00:17:10.347>as <00:17:10.571>well.

292
00:17:10.802 --> 00:17:14.583
<00:17:10.801>So <00:17:10.901>both <00:17:11.002>of <00:17:11.101>these <00:17:11.202>weird <00:17:11.301>bits <00:17:11.401>that <00:17:11.502>we <00:17:11.601>had <00:17:11.702>in <00:17:11.801>chat, <00:17:11.901>do <00:17:12.002>you <00:17:12.101>see <00:17:12.202>them

293
00:17:14.642 --> 00:17:16.482
<00:17:14.642>happening <00:17:14.842>in <00:17:15.042>an <00:17:15.242>agentic <00:17:15.442>world <00:17:15.642>now <00:17:15.842>as <00:17:16.042>well?

294
00:17:17.282 --> 00:17:20.562
<00:17:17.281>I <00:17:17.482>see <00:17:17.682>them <00:17:17.881>happening. <00:17:18.081>I <00:17:18.281>can <00:17:18.482>see <00:17:18.682>that

295
00:17:22.062 --> 00:17:25.843
<00:17:22.061>there <00:17:22.217>are <00:17:22.373>ways, <00:17:22.528>and <00:17:22.684>the <00:17:22.839>latest <00:17:22.995>models, <00:17:23.150>especially <00:17:23.306>the <00:17:23.462>state-of-the-art

296
00:17:25.902 --> 00:17:29.812
<00:17:25.902>models, <00:17:26.182>are <00:17:26.462>trying <00:17:26.742>to <00:17:27.021>find <00:17:27.302>interesting

297
00:17:29.862 --> 00:17:33.402
<00:17:29.862>ways <00:17:29.989>at <00:17:30.116>model <00:17:30.243>level, <00:17:30.371>and <00:17:30.498>then <00:17:30.625>we <00:17:30.752>are <00:17:30.880>adding <00:17:31.007>guardrails <00:17:31.134>at <00:17:31.262>the

298
00:17:33.422 --> 00:17:37.062
<00:17:33.422>agentic <00:17:33.622>level, <00:17:33.822>at <00:17:34.022>the <00:17:34.222>agent <00:17:34.422>level, <00:17:34.622>to <00:17:34.822>let's

299
00:17:37.082 --> 00:17:39.262
<00:17:37.082>say <00:17:37.362>reduce <00:17:37.642>the <00:17:37.922>number <00:17:38.202>of <00:17:38.482>hallucinations.

300
00:17:39.322 --> 00:17:41.202
<00:17:39.322>In <00:17:39.462>fact, <00:17:39.602>there <00:17:39.742>are <00:17:39.882>also <00:17:40.022>a <00:17:40.162>couple <00:17:40.302>of <00:17:40.442>benchmarks <00:17:40.582>out <00:17:40.722>there.

301
00:17:42.962 --> 00:17:46.702
<00:17:42.962>And <00:17:43.055>then <00:17:43.148>at <00:17:43.242>the <00:17:43.335>system <00:17:43.428>level, <00:17:43.522>so <00:17:43.615>let's <00:17:43.708>assume <00:17:43.802>that <00:17:43.895>the <00:17:43.988>agent <00:17:44.081>is <00:17:44.175>still <00:17:44.268>able <00:17:44.362>to

302
00:17:46.722 --> 00:17:50.602
<00:17:46.722>hallucinate <00:17:46.822>quite <00:17:46.922>a <00:17:47.022>bit, <00:17:47.122>right? <00:17:47.222>And <00:17:47.322>then <00:17:47.422>at <00:17:47.522>the <00:17:47.622>system <00:17:47.722>level, <00:17:47.822>we <00:17:47.922>are <00:17:48.022>trying <00:17:48.122>to

303
00:17:50.642 --> 00:17:54.442
<00:17:50.642>block <00:17:50.875>everything <00:17:51.108>towards <00:17:51.342>guardrails <00:17:51.575>to <00:17:51.808>restrict <00:17:52.042>the

304
00:17:54.482 --> 00:17:55.582
<00:17:54.482>access <00:17:54.680>as <00:17:54.878>much <00:17:55.076>as <00:17:55.274>we <00:17:55.472>can.

305
00:17:57.222 --> 00:18:01.072
<00:17:57.222>That's <00:17:57.349>something <00:17:57.476>that <00:17:57.603>we <00:17:57.731>did <00:17:57.858>since <00:17:57.985>the <00:17:58.112>start <00:17:58.240>of <00:17:58.367>computer <00:17:58.494>science, <00:17:58.622>right?

306
00:18:01.122 --> 00:18:03.442
<00:18:01.122>So <00:18:01.277>just <00:18:01.433>block <00:18:01.588>the <00:18:01.744>access <00:18:01.899>as <00:18:02.055>much <00:18:02.210>as <00:18:02.366>you <00:18:02.522>can.

307
00:18:04.102 --> 00:18:06.502
<00:18:04.102>But <00:18:04.302>nowadays, <00:18:04.502>seems <00:18:04.702>like <00:18:04.902>people <00:18:05.102>forgot <00:18:05.302>about <00:18:05.502>it.

308
00:18:08.022 --> 00:18:10.702
<00:18:08.022>But <00:18:08.255>yeah. <00:18:08.488>I <00:18:08.722>think <00:18:08.955>it's <00:18:09.188>still <00:18:09.422>happening.

309
00:18:11.802 --> 00:18:15.482
<00:18:11.802>Since <00:18:11.918>the <00:18:12.035>start, <00:18:12.151>I <00:18:12.268>think <00:18:12.385>we <00:18:12.502>are <00:18:12.618>finding <00:18:12.735>new <00:18:12.851>techniques <00:18:12.968>to <00:18:13.085>mitigate <00:18:13.202>it.

310
00:18:16.162 --> 00:18:20.042
<00:18:16.162>The <00:18:16.302>interesting <00:18:16.442>bit <00:18:16.582>about <00:18:16.722>this <00:18:16.862>one <00:18:17.002>as <00:18:17.142>well <00:18:17.282>is <00:18:17.422>when <00:18:17.562>you

311
00:18:20.102 --> 00:18:23.942
<00:18:20.102>see <00:18:20.302>the <00:18:20.502>supply <00:18:20.702>chain <00:18:20.902>attacks <00:18:21.102>that <00:18:21.302>we <00:18:21.502>have

312
00:18:23.962 --> 00:18:27.722
<00:18:23.962>nowadays, <00:18:24.117>like <00:18:24.273>systems <00:18:24.428>come <00:18:24.584>up <00:18:24.739>with <00:18:24.895>NPM <00:18:25.050>modules <00:18:25.206>that <00:18:25.362>don't

313
00:18:27.762 --> 00:18:31.602
<00:18:27.762>exist, <00:18:27.917>and <00:18:28.073>then <00:18:28.228>malware <00:18:28.384>creators <00:18:28.539>actually <00:18:28.695>create <00:18:28.850>those <00:18:29.006>things <00:18:29.162>because

314
00:18:31.642 --> 00:18:34.002
<00:18:31.642>they <00:18:31.875>saw <00:18:32.108>an <00:18:32.342>LLM <00:18:32.575>actually <00:18:32.808>hallucinating <00:18:33.042>it.

315
00:18:34.402 --> 00:18:37.402
<00:18:34.402>So <00:18:34.518>it <00:18:34.635>seems <00:18:34.752>to <00:18:34.868>be <00:18:34.985>an <00:18:35.102>arms <00:18:35.218>race <00:18:35.335>in <00:18:35.452>this <00:18:35.568>kind <00:18:35.685>of <00:18:35.802>environment.

316
00:18:37.902 --> 00:18:41.542
<00:18:37.902>Now, <00:18:38.029>you <00:18:38.156>talked <00:18:38.283>about <00:18:38.411>in <00:18:38.538>your <00:18:38.665>proposal <00:18:38.792>that <00:18:38.920>you're <00:18:39.047>talking <00:18:39.174>about <00:18:39.302>real-world

317
00:18:41.582 --> 00:18:43.442
<00:18:41.582>examples. <00:18:42.048>Is <00:18:42.515>there <00:18:42.982>any

318
00:18:44.422 --> 00:18:48.102
<00:18:44.422>example <00:18:44.509>that <00:18:44.597>you <00:18:44.684>did <00:18:44.772>for <00:18:44.859>a <00:18:44.947>client <00:18:45.034>or <00:18:45.122>did <00:18:45.209>with <00:18:45.297>a <00:18:45.384>client <00:18:45.472>where <00:18:45.559>they <00:18:45.647>actually <00:18:45.734>built <00:18:45.822>an

319
00:18:48.182 --> 00:18:51.062
<00:18:48.182>agent, <00:18:48.289>and <00:18:48.397>what <00:18:48.505>did <00:18:48.612>they <00:18:48.720>use <00:18:48.828>it <00:18:48.935>for, <00:18:49.043>and <00:18:49.151>what <00:18:49.258>was <00:18:49.366>the <00:18:49.474>success <00:18:49.582>rate?

320
00:18:51.566 --> 00:18:55.206
<00:18:51.566>Maybe <00:18:51.721>I <00:18:51.877>can <00:18:52.032>give <00:18:52.188>you <00:18:52.343>a <00:18:52.499>very <00:18:52.654>brief <00:18:52.810>overview <00:18:52.966>about

321
00:18:55.526 --> 00:18:59.426
<00:18:55.526>an <00:18:55.642>agentic <00:18:55.759>system <00:18:55.876>that <00:18:55.992>we <00:18:56.109>developed <00:18:56.226>and <00:18:56.342>that <00:18:56.459>we <00:18:56.576>are <00:18:56.692>developing, <00:18:56.809>if <00:18:56.926>that's

322
00:18:59.466 --> 00:18:59.725
fine.

323
00:19:00.166 --> 00:19:00.316
Yeah.

324
00:19:00.486 --> 00:19:03.786
<00:19:00.486>And <00:19:00.661>maybe <00:19:00.836>just <00:19:01.011>sketch <00:19:01.186>out <00:19:01.361>very <00:19:01.536>quickly <00:19:01.711>some <00:19:01.886>of

325
00:19:03.866 --> 00:19:06.546
<00:19:03.866>the <00:19:04.146>interesting <00:19:04.426>learnings <00:19:04.706>that <00:19:04.985>we <00:19:05.266>found.

326
00:19:06.586 --> 00:19:06.706
Yeah.

327
00:19:07.206 --> 00:19:11.006
<00:19:07.206>Especially <00:19:07.406>while <00:19:07.606>trying <00:19:07.805>to <00:19:08.005>make <00:19:08.206>this <00:19:08.406>system <00:19:08.606>reliable,

328
00:19:11.696 --> 00:19:13.586
<00:19:11.696>which <00:19:11.929>is, <00:19:12.162>I <00:19:12.396>think, <00:19:12.629>the <00:19:12.862>interesting <00:19:13.096>part.

329
00:19:13.666 --> 00:19:17.106
<00:19:13.666>So <00:19:13.866>one <00:19:14.066>of <00:19:14.265>the <00:19:14.466>things <00:19:14.666>that <00:19:14.866>we <00:19:15.066>underestimated

330
00:19:17.146 --> 00:19:20.986
<00:19:17.146>initially <00:19:17.321>is <00:19:17.495>how <00:19:17.671>much <00:19:17.846>is <00:19:18.021>important, <00:19:18.196>and <00:19:18.370>how <00:19:18.546>much

331
00:19:21.146 --> 00:19:24.946
<00:19:21.146>time <00:19:21.321>and <00:19:21.495>effort <00:19:21.671>it <00:19:21.846>takes <00:19:22.021>to <00:19:22.196>build <00:19:22.370>a <00:19:22.546>proper

332
00:19:24.966 --> 00:19:28.535
<00:19:24.966>golden <00:19:25.245>data <00:19:25.525>set, <00:19:25.805>with <00:19:26.085>enough <00:19:26.366>examples

333
00:19:28.926 --> 00:19:31.266
<00:19:28.926>and <00:19:29.275>with <00:19:29.626>enough <00:19:29.976>different <00:19:30.326>examples.

334
00:19:31.416 --> 00:19:34.846
<00:19:31.416>So <00:19:31.523>what <00:19:31.631>we <00:19:31.739>want <00:19:31.846>in <00:19:31.954>the <00:19:32.062>golden <00:19:32.169>data <00:19:32.277>set <00:19:32.385>is <00:19:32.492>to <00:19:32.600>have <00:19:32.708>a <00:19:32.816>wide

335
00:19:34.926 --> 00:19:38.326
<00:19:34.926>distribution <00:19:35.081>of <00:19:35.237>what <00:19:35.392>we <00:19:35.548>believe <00:19:35.703>user <00:19:35.859>queries <00:19:36.014>might <00:19:36.170>be. <00:19:36.326>Right?

336
00:19:38.946 --> 00:19:40.746
<00:19:38.946>And <00:19:39.179>as <00:19:39.412>soon <00:19:39.646>as <00:19:39.879>we <00:19:40.112>have <00:19:40.346>data

337
00:19:41.926 --> 00:19:45.406
<00:19:41.926>coming <00:19:42.101>from, <00:19:42.275>let's <00:19:42.451>say <00:19:42.626>that <00:19:42.801>we <00:19:42.976>are <00:19:43.150>using <00:19:43.326>Lengthmeter,

338
00:19:45.486 --> 00:19:49.026
<00:19:45.486>Longfuse, <00:19:45.766>or <00:19:46.046>whatever <00:19:46.326>observability <00:19:46.606>tool <00:19:46.886>to

339
00:19:49.146 --> 00:19:52.786
<00:19:49.146>track <00:19:49.253>our <00:19:49.361>user <00:19:49.469>interactions, <00:19:49.576>then <00:19:49.684>what <00:19:49.792>we <00:19:49.899>can <00:19:50.007>do <00:19:50.115>is, <00:19:50.222>as <00:19:50.330>soon <00:19:50.438>as <00:19:50.546>we

340
00:19:52.826 --> 00:19:56.676
<00:19:52.826>recognize <00:19:52.953>a <00:19:53.080>new <00:19:53.207>failure <00:19:53.335>mode <00:19:53.462>which <00:19:53.589>we <00:19:53.716>didn't <00:19:53.844>address <00:19:53.971>yet, <00:19:54.098>we <00:19:54.226>take

341
00:19:56.746 --> 00:20:00.586
<00:19:56.746>that, <00:19:56.862>we <00:19:56.979>put <00:19:57.096>it <00:19:57.212>into <00:19:57.329>the <00:19:57.446>golden <00:19:57.562>data <00:19:57.679>set, <00:19:57.796>and <00:19:57.912>then <00:19:58.029>we <00:19:58.146>try

342
00:20:00.706 --> 00:20:04.586
<00:20:00.706>to <00:20:00.833>fix <00:20:00.960>our <00:20:01.087>evals <00:20:01.215>by <00:20:01.342>changing <00:20:01.469>the <00:20:01.596>behavior <00:20:01.724>of <00:20:01.851>the <00:20:01.978>system, <00:20:02.106>if

343
00:20:04.646 --> 00:20:07.606
<00:20:04.646>there <00:20:04.786>is <00:20:04.926>a <00:20:05.066>failure, <00:20:05.206>of <00:20:05.346>course. <00:20:05.485>So <00:20:05.626>that's <00:20:05.765>the <00:20:05.906>first <00:20:06.046>thing.

344
00:20:08.486 --> 00:20:10.946
<00:20:08.486>The <00:20:08.641>second <00:20:08.797>thing, <00:20:08.952>still <00:20:09.108>related <00:20:09.263>to <00:20:09.419>the <00:20:09.574>golden <00:20:09.730>data <00:20:09.886>set,

345
00:20:11.766 --> 00:20:12.946
<00:20:11.766>which <00:20:12.031>we <00:20:12.297>found <00:20:12.562>out, <00:20:12.828>is

346
00:20:13.966 --> 00:20:14.766
<00:20:13.966>how <00:20:14.686>much

347
00:20:15.606 --> 00:20:19.406
<00:20:15.606>deep <00:20:15.806>connection <00:20:16.006>the <00:20:16.206>engineering <00:20:16.406>team <00:20:16.606>and <00:20:16.806>the <00:20:17.006>subject

348
00:20:19.466 --> 00:20:21.386
<00:20:19.466>matter <00:20:19.745>experts <00:20:20.025>team <00:20:20.305>needs <00:20:20.585>to <00:20:20.866>have.

349
00:20:21.926 --> 00:20:24.856
<00:20:21.926>So <00:20:22.081>usually <00:20:22.237>when <00:20:22.392>you're <00:20:22.548>building, <00:20:22.703>let's <00:20:22.859>say, <00:20:23.014>a <00:20:23.170>traditional <00:20:23.326>system,

350
00:20:26.006 --> 00:20:29.926
<00:20:26.006>you <00:20:26.146>can <00:20:26.286>just <00:20:26.426>ask <00:20:26.566>to <00:20:26.706>the <00:20:26.846>subject <00:20:26.986>matter <00:20:27.126>experts <00:20:27.266>once, <00:20:27.406>"Hey,

351
00:20:30.126 --> 00:20:33.746
<00:20:30.126>please <00:20:30.242>give <00:20:30.359>me <00:20:30.476>your <00:20:30.592>requirements." <00:20:30.709>You <00:20:30.826>get <00:20:30.942>a <00:20:31.059>list <00:20:31.176>of <00:20:31.292>requirements, <00:20:31.409>you <00:20:31.526>implement

352
00:20:33.806 --> 00:20:35.366
<00:20:33.806>them, <00:20:34.086>and <00:20:34.366>then <00:20:34.646>you <00:20:34.926>come <00:20:35.206>back.

353
00:20:36.826 --> 00:20:40.486
<00:20:36.826>But <00:20:37.001>again, <00:20:37.176>when <00:20:37.351>the <00:20:37.526>input <00:20:37.701>is <00:20:37.876>non-deterministic, <00:20:38.051>when <00:20:38.226>the

354
00:20:40.506 --> 00:20:44.066
<00:20:40.506>input <00:20:40.681>is <00:20:40.856>a <00:20:41.031>natural <00:20:41.206>language <00:20:41.381>query <00:20:41.556>from <00:20:41.731>a <00:20:41.906>user,

355
00:20:44.226 --> 00:20:48.086
<00:20:44.226>you <00:20:44.353>cannot <00:20:44.480>do <00:20:44.607>that <00:20:44.735>in <00:20:44.862>advance. <00:20:44.989>So <00:20:45.116>you <00:20:45.244>have <00:20:45.371>to <00:20:45.498>work <00:20:45.626>a

356
00:20:48.226 --> 00:20:51.146
<00:20:48.226>lot <00:20:48.381>with <00:20:48.537>the <00:20:48.692>subject <00:20:48.848>matter <00:20:49.003>experts <00:20:49.159>to <00:20:49.314>come <00:20:49.470>up <00:20:49.626>with

357
00:20:51.966 --> 00:20:55.686
<00:20:51.966>the <00:20:52.121>possible <00:20:52.277>range <00:20:52.432>of <00:20:52.588>questions, <00:20:52.743>and <00:20:52.899>you <00:20:53.054>have <00:20:53.210>to <00:20:53.366>optimize

358
00:20:55.786 --> 00:20:56.006
for

359
00:20:56.986 --> 00:20:59.486
<00:20:56.986>the <00:20:57.126>answers. <00:20:57.266>What <00:20:57.406>is <00:20:57.546>the <00:20:57.686>tools <00:20:57.826>that <00:20:57.966>we <00:20:58.106>want <00:20:58.246>to <00:20:58.386>use?

360
00:21:01.046 --> 00:21:04.346
<00:21:01.046>So <00:21:01.201>all <00:21:01.357>these <00:21:01.512>things <00:21:01.668>are <00:21:01.823>very <00:21:01.979>important <00:21:02.134>and <00:21:02.290>very <00:21:02.446>interesting.

361
00:21:05.326 --> 00:21:08.886
<00:21:05.326>And <00:21:05.501>then <00:21:05.676>something <00:21:05.851>interesting <00:21:06.026>that <00:21:06.201>we <00:21:06.376>noticed <00:21:06.551>is <00:21:06.726>also

362
00:21:08.986 --> 00:21:12.746
<00:21:08.986>collecting <00:21:09.452>feedback <00:21:09.919>from <00:21:10.386>real

363
00:21:12.786 --> 00:21:16.606
<00:21:12.786>users, <00:21:13.066>plus <00:21:13.346>subject <00:21:13.626>matter <00:21:13.906>experts' <00:21:14.186>annotations

364
00:21:17.066 --> 00:21:18.446
<00:21:17.066>on <00:21:17.480>the <00:21:17.894>tracing <00:21:18.308>system

365
00:21:20.026 --> 00:21:20.806
<00:21:20.026>is <00:21:20.728>what

366
00:21:21.926 --> 00:21:25.846
<00:21:21.926>gives <00:21:22.126>us <00:21:22.326>enough <00:21:22.525>information <00:21:22.726>to <00:21:22.926>improve <00:21:23.126>the <00:21:23.326>system,

367
00:21:26.546 --> 00:21:29.766
<00:21:26.546>not <00:21:26.721>autonomously, <00:21:26.896>but <00:21:27.071>with <00:21:27.246>a <00:21:27.421>very <00:21:27.596>good <00:21:27.771>feedback <00:21:27.946>loop.

368
00:21:30.166 --> 00:21:33.846
<00:21:30.166>So <00:21:30.321>basically, <00:21:30.477>the <00:21:30.632>user, <00:21:30.788>with <00:21:30.943>usual <00:21:31.099>thumbs <00:21:31.254>up, <00:21:31.410>thumbs <00:21:31.566>down,

369
00:21:33.966 --> 00:21:37.826
<00:21:33.966>and <00:21:34.121>optional <00:21:34.277>comment, <00:21:34.432>can <00:21:34.588>say, <00:21:34.743>"Okay, <00:21:34.899>that's <00:21:35.054>what <00:21:35.210>I <00:21:35.366>was

370
00:21:37.866 --> 00:21:41.746
<00:21:37.866>expecting. <00:21:37.973>That's <00:21:38.081>good. <00:21:38.189>That's <00:21:38.296>wrong." <00:21:38.404>Then <00:21:38.512>we <00:21:38.619>use <00:21:38.727>that <00:21:38.835>as, <00:21:38.942>let's <00:21:39.050>say, <00:21:39.158>an <00:21:39.266>alert

371
00:21:42.386 --> 00:21:46.116
<00:21:42.386>for <00:21:42.513>our <00:21:42.640>subject <00:21:42.767>matter <00:21:42.895>experts <00:21:43.022>to <00:21:43.149>annotate <00:21:43.276>the <00:21:43.404>trace <00:21:43.531>and <00:21:43.658>to <00:21:43.786>say,

372
00:21:47.106 --> 00:21:51.026
<00:21:47.106>"Hey, <00:21:47.233>that's <00:21:47.360>the <00:21:47.487>output. <00:21:47.615>The <00:21:47.742>output <00:21:47.869>went <00:21:47.996>wrong <00:21:48.124>here <00:21:48.251>because <00:21:48.378>maybe, <00:21:48.506>I

373
00:21:51.066 --> 00:21:54.266
<00:21:51.066>don't <00:21:51.159>know, <00:21:51.252>this <00:21:51.346>tool <00:21:51.439>has <00:21:51.532>not <00:21:51.626>been <00:21:51.719>called, <00:21:51.812>or <00:21:51.906>this <00:21:51.999>tool <00:21:52.092>has <00:21:52.186>been <00:21:52.279>called <00:21:52.372>with <00:21:52.466>the

374
00:21:54.286 --> 00:21:57.146
<00:21:54.286>wrong <00:21:54.441>parameters, <00:21:54.597>or <00:21:54.752>I <00:21:54.908>didn't <00:21:55.063>find <00:21:55.219>the <00:21:55.374>right <00:21:55.530>information." <00:21:55.686>Right?

375
00:21:58.566 --> 00:22:02.046
<00:21:58.566>And <00:21:58.693>then <00:21:58.820>we <00:21:58.947>use <00:21:59.075>that, <00:21:59.202>again, <00:21:59.329>in <00:21:59.456>feedback <00:21:59.584>loop <00:21:59.711>to <00:21:59.838>improve <00:21:59.966>the

376
00:22:02.666 --> 00:22:02.966
agents.

377
00:22:04.346 --> 00:22:07.806
<00:22:04.346>Last <00:22:04.473>bit <00:22:04.600>on <00:22:04.727>this, <00:22:04.855>on <00:22:04.982>the <00:22:05.109>golden <00:22:05.236>data <00:22:05.364>set, <00:22:05.491>which <00:22:05.618>we <00:22:05.746>found

378
00:22:07.886 --> 00:22:09.406
<00:22:07.886>interesting, <00:22:09.254>is

379
00:22:10.406 --> 00:22:11.826
<00:22:10.406>we <00:22:10.831>run <00:22:11.258>a <00:22:11.684>loop

380
00:22:12.746 --> 00:22:16.106
<00:22:12.746>to <00:22:12.921>automate <00:22:13.096>the <00:22:13.271>evals <00:22:13.446>and <00:22:13.621>to <00:22:13.796>fix <00:22:13.971>the <00:22:14.146>system.

381
00:22:16.186 --> 00:22:20.086
<00:22:16.186>So <00:22:16.341>basically, <00:22:16.497>we <00:22:16.652>developed <00:22:16.808>a <00:22:16.963>very <00:22:17.119>small <00:22:17.274>tool, <00:22:17.430>which <00:22:17.586>allows

382
00:22:20.106 --> 00:22:24.026
<00:22:20.106>us <00:22:20.246>to <00:22:20.386>fix <00:22:20.526>some <00:22:20.666>of <00:22:20.806>the <00:22:20.946>issues <00:22:21.086>of <00:22:21.226>the <00:22:21.366>system <00:22:21.506>almost

383
00:22:24.046 --> 00:22:28.026
<00:22:24.046>autonomously. <00:22:24.221>So <00:22:24.396>it <00:22:24.571>basically <00:22:24.746>generates <00:22:24.921>some <00:22:25.096>candidates <00:22:25.271>for <00:22:25.446>improvements,

384
00:22:28.106 --> 00:22:31.166
<00:22:28.106>and <00:22:28.261>then <00:22:28.417>we <00:22:28.572>review <00:22:28.728>them, <00:22:28.883>accept <00:22:29.039>them, <00:22:29.194>or <00:22:29.350>reject <00:22:29.506>them.

385
00:22:32.186 --> 00:22:35.886
<00:22:32.186>And <00:22:32.326>we <00:22:32.466>noticed <00:22:32.606>that <00:22:32.745>in <00:22:32.886>some <00:22:33.025>cases, <00:22:33.166>the <00:22:33.305>system <00:22:33.446>itself <00:22:33.586>found

386
00:22:36.006 --> 00:22:39.226
<00:22:36.006>issues <00:22:36.206>in <00:22:36.406>the <00:22:36.606>upstream <00:22:36.806>data. <00:22:37.006>So <00:22:37.206>the <00:22:37.406>actual

387
00:22:39.846 --> 00:22:43.786
<00:22:39.846>system <00:22:40.046>response <00:22:40.246>was <00:22:40.446>kind <00:22:40.646>of <00:22:40.846>correct, <00:22:41.046>but <00:22:41.246>the

388
00:22:43.846 --> 00:22:47.566
<00:22:43.846>upstream <00:22:44.021>data <00:22:44.196>was <00:22:44.371>wrong. <00:22:44.546>So <00:22:44.721>it's <00:22:44.896>interesting <00:22:45.071>that <00:22:45.246>while

389
00:22:47.606 --> 00:22:51.386
<00:22:47.606>analyzing <00:22:47.781>the <00:22:47.956>behavior <00:22:48.131>of <00:22:48.306>AI <00:22:48.481>agents, <00:22:48.656>we <00:22:48.831>can <00:22:49.006>also

390
00:22:51.526 --> 00:22:55.315
<00:22:51.526>spot <00:22:51.806>issues <00:22:52.086>and, <00:22:52.366>let's <00:22:52.646>say, <00:22:52.926>bugs,

391
00:22:55.526 --> 00:22:58.466
<00:22:55.526>problems <00:22:55.681>in <00:22:55.837>other <00:22:55.992>parts <00:22:56.148>of <00:22:56.303>the <00:22:56.459>system <00:22:56.614>which <00:22:56.770>are <00:22:56.926>connected.

392
00:22:58.966 --> 00:23:02.026
<00:22:58.966>In <00:22:59.106>essence, <00:22:59.245>it's <00:22:59.386>just <00:22:59.525>another <00:22:59.666>user, <00:22:59.805>isn't <00:22:59.946>it, <00:23:00.085>like <00:23:00.226>an <00:23:00.366>agent?

393
00:23:02.166 --> 00:23:04.926
<00:23:02.166>It's <00:23:02.341>basically, <00:23:02.515>that's <00:23:02.691>why <00:23:02.866>you <00:23:03.041>find <00:23:03.216>problems <00:23:03.390>in <00:23:03.566>the

394
00:23:05.646 --> 00:23:08.546
<00:23:05.646>surrounding <00:23:05.879>plumbing <00:23:06.112>of <00:23:06.346>your <00:23:06.579>application <00:23:06.812>as <00:23:07.046>well.

395
00:23:09.686 --> 00:23:13.526
<00:23:09.686>It <00:23:09.785>reminded <00:23:09.886>me <00:23:09.985>when <00:23:10.086>you <00:23:10.186>just <00:23:10.285>said <00:23:10.386>about <00:23:10.485>the <00:23:10.586>things <00:23:10.686>that <00:23:10.785>you <00:23:10.886>do, <00:23:10.985>I <00:23:11.086>remember

396
00:23:13.986 --> 00:23:16.126
<00:23:13.986>FAQ <00:23:14.266>systems <00:23:14.546>on <00:23:14.826>websites <00:23:15.106>back <00:23:15.386>then.

397
00:23:16.186 --> 00:23:19.906
<00:23:16.186>You <00:23:16.326>basically <00:23:16.466>set <00:23:16.606>them <00:23:16.745>up <00:23:16.886>with <00:23:17.025>the <00:23:17.166>customer, <00:23:17.305>and <00:23:17.446>here's <00:23:17.586>the

398
00:23:19.966 --> 00:23:22.066
<00:23:19.966>questions <00:23:20.166>that <00:23:20.366>you <00:23:20.565>probably <00:23:20.765>get <00:23:20.966>all <00:23:21.166>the <00:23:21.366>time.

399
00:23:22.466 --> 00:23:26.226
<00:23:22.466>We <00:23:22.565>put <00:23:22.666>those <00:23:22.765>in, <00:23:22.866>and <00:23:22.966>then <00:23:23.065>you <00:23:23.166>basically <00:23:23.265>show <00:23:23.366>people <00:23:23.466>the <00:23:23.565>answers <00:23:23.666>already <00:23:23.765>when <00:23:23.866>they

400
00:23:26.306 --> 00:23:29.226
<00:23:26.306>ask <00:23:26.433>a <00:23:26.560>question <00:23:26.687>that <00:23:26.815>is <00:23:26.942>similar. <00:23:27.069>And <00:23:27.196>that <00:23:27.324>was <00:23:27.451>the <00:23:27.578>first <00:23:27.706>LLM,

401
00:23:30.086 --> 00:23:32.246
<00:23:30.086>natural <00:23:30.286>language <00:23:30.486>processing <00:23:30.686>things <00:23:30.886>that <00:23:31.086>I <00:23:31.286>worked <00:23:31.486>with.

402
00:23:32.716 --> 00:23:36.526
<00:23:32.716>And <00:23:32.832>basically <00:23:32.949>then <00:23:33.065>have <00:23:33.182>an <00:23:33.299>FAQ <00:23:33.416>that <00:23:33.532>grows <00:23:33.649>with <00:23:33.765>the <00:23:33.882>questions <00:23:33.999>of <00:23:34.116>people.

403
00:23:37.166 --> 00:23:40.586
<00:23:37.166>And <00:23:37.259>it <00:23:37.352>feels <00:23:37.446>like <00:23:37.539>we're <00:23:37.632>doing <00:23:37.726>the <00:23:37.819>same <00:23:37.912>thing <00:23:38.005>now <00:23:38.099>when <00:23:38.192>we're <00:23:38.285>setting <00:23:38.379>up <00:23:38.472>the <00:23:38.566>system

404
00:23:40.666 --> 00:23:42.766
<00:23:40.666>prompt, <00:23:40.841>and <00:23:41.015>we're <00:23:41.191>setting <00:23:41.366>up <00:23:41.541>the <00:23:41.716>gold <00:23:41.890>data <00:23:42.066>set.

405
00:23:43.326 --> 00:23:47.146
<00:23:43.326>And <00:23:43.466>it <00:23:43.606>fascinates <00:23:43.746>me <00:23:43.886>when <00:23:44.026>I <00:23:44.166>see <00:23:44.306>system <00:23:44.446>prompt <00:23:44.586>demos <00:23:44.726>that

406
00:23:47.206 --> 00:23:51.106
<00:23:47.206>always <00:23:47.346>says, <00:23:47.485>"Don't <00:23:47.626>make <00:23:47.765>mistakes." <00:23:47.906>It <00:23:48.045>always <00:23:48.186>fascinates <00:23:48.325>me <00:23:48.466>that <00:23:48.606>we

407
00:23:51.226 --> 00:23:54.206
<00:23:51.226>have <00:23:51.366>to <00:23:51.506>tell <00:23:51.646>the <00:23:51.786>machine <00:23:51.926>that. <00:23:52.066>Shouldn't <00:23:52.206>that <00:23:52.346>be <00:23:52.486>a <00:23:52.626>preset?

408
00:23:55.706 --> 00:23:58.086
<00:23:55.706>Yeah, <00:23:56.172>that's <00:23:56.639>interesting. <00:23:57.106>It's

409
00:23:58.926 --> 00:24:02.126
<00:23:58.926>one <00:23:59.081>of <00:23:59.237>the <00:23:59.392>cool <00:23:59.548>things <00:23:59.703>of <00:23:59.859>working <00:24:00.014>with <00:24:00.170>LLMs, <00:24:00.326>so.

410
00:24:02.946 --> 00:24:06.886
<00:24:02.946>But <00:24:03.028>back <00:24:03.110>in <00:24:03.193>the <00:24:03.275>days, <00:24:03.357>which <00:24:03.440>is <00:24:03.522>two <00:24:03.604>years <00:24:03.687>ago, <00:24:03.769>but <00:24:03.851>we <00:24:03.934>can <00:24:04.016>say <00:24:04.098>back <00:24:04.181>in <00:24:04.263>the <00:24:04.346>days,

411
00:24:07.506 --> 00:24:11.136
<00:24:07.506>you <00:24:07.681>had <00:24:07.856>to <00:24:08.031>add <00:24:08.206>in <00:24:08.381>the <00:24:08.556>prompt <00:24:08.731>all <00:24:08.906>these

412
00:24:11.186 --> 00:24:14.936
<00:24:11.186>interesting <00:24:11.386>things. <00:24:11.586>Like <00:24:11.785>the <00:24:11.985>CoT, <00:24:12.186>for <00:24:12.386>example, <00:24:12.586>is

413
00:24:14.966 --> 00:24:17.086
<00:24:14.966>born <00:24:15.245>from <00:24:15.525>a <00:24:15.805>clever <00:24:16.085>prompt, <00:24:16.366>right?

414
00:24:18.366 --> 00:24:19.606
<00:24:18.366>And <00:24:18.552>so <00:24:18.738>there <00:24:18.924>are <00:24:19.110>all <00:24:19.296>these <00:24:19.482>things.

415
00:24:20.526 --> 00:24:24.426
<00:24:20.526>Luckily <00:24:20.681>for <00:24:20.837>us, <00:24:20.992>the <00:24:21.148>model <00:24:21.303>providers <00:24:21.459>nowadays <00:24:21.614>are <00:24:21.770>doing <00:24:21.926>a

416
00:24:24.506 --> 00:24:27.856
<00:24:24.506>lot <00:24:24.706>of <00:24:24.906>these <00:24:25.106>things <00:24:25.306>in <00:24:25.506>the <00:24:25.706>pre-training <00:24:25.906>and

417
00:24:27.926 --> 00:24:29.486
<00:24:27.926>post-training <00:24:29.326>phases.

418
00:24:30.426 --> 00:24:33.726
<00:24:30.426>So <00:24:30.581>for <00:24:30.737>example, <00:24:30.892>the <00:24:31.048>CoT <00:24:31.203>is <00:24:31.359>done <00:24:31.514>automatically <00:24:31.670>by <00:24:31.826>the

419
00:24:34.626 --> 00:24:38.546
<00:24:34.626>model <00:24:34.753>providers. <00:24:34.880>We <00:24:35.007>are <00:24:35.135>doing <00:24:35.262>a <00:24:35.389>lot <00:24:35.516>of <00:24:35.644>reinforcement <00:24:35.771>learning <00:24:35.898>on <00:24:36.026>interesting

420
00:24:38.626 --> 00:24:40.466
<00:24:38.626>things <00:24:38.906>that <00:24:39.186>we <00:24:39.466>want <00:24:39.745>to <00:24:40.026>fine-tune.

421
00:24:41.366 --> 00:24:45.306
<00:24:41.366>But <00:24:41.506>yeah, <00:24:41.646>we <00:24:41.786>still <00:24:41.926>find <00:24:42.066>out <00:24:42.206>these <00:24:42.346>days <00:24:42.485>that <00:24:42.626>in <00:24:42.766>some

422
00:24:45.366 --> 00:24:46.406
<00:24:45.366>cases, <00:24:45.834>adding <00:24:46.302>some

423
00:24:47.646 --> 00:24:48.726
<00:24:47.646>simple <00:24:48.618>sentences,

424
00:24:49.746 --> 00:24:53.026
<00:24:49.746>like <00:24:49.886>the <00:24:50.026>one <00:24:50.166>that <00:24:50.306>you <00:24:50.446>mentioned, <00:24:50.586>or <00:24:50.726>simple <00:24:50.866>sentences <00:24:51.006>which <00:24:51.146>are

425
00:24:53.046 --> 00:24:56.925
<00:24:53.046>domain-specific, <00:24:53.246>which <00:24:53.446>are <00:24:53.646>specific <00:24:53.846>for <00:24:54.046>an <00:24:54.246>agent, <00:24:54.446>can

426
00:24:57.026 --> 00:25:00.886
<00:24:57.026>bump <00:24:57.181>the <00:24:57.337>evals <00:24:57.492>by <00:24:57.648>quite <00:24:57.803>a <00:24:57.959>bit <00:24:58.114>and <00:24:58.270>can <00:24:58.426>just

427
00:25:00.946 --> 00:25:04.826
<00:25:00.946>remove <00:25:01.062>an <00:25:01.179>entire <00:25:01.295>set <00:25:01.412>of <00:25:01.529>failure <00:25:01.646>modes <00:25:01.762>with <00:25:01.879>just <00:25:01.995>one <00:25:02.112>sentence <00:25:02.229>in <00:25:02.346>the

428
00:25:04.846 --> 00:25:06.656
<00:25:04.846>system <00:25:05.196>prompt, <00:25:05.546>which <00:25:05.896>is <00:25:06.246>incredible.

429
00:25:07.270 --> 00:25:09.830
<00:25:07.270>What <00:25:07.470>use <00:25:07.670>cases <00:25:07.870>do <00:25:08.070>you <00:25:08.270>see <00:25:08.470>for <00:25:08.670>agents?

430
00:25:09.910 --> 00:25:13.490
<00:25:09.910>The <00:25:10.065>people <00:25:10.221>that <00:25:10.376>you <00:25:10.532>work <00:25:10.687>with, <00:25:10.843>why <00:25:10.998>are <00:25:11.154>they <00:25:11.310>building

431
00:25:13.590 --> 00:25:17.410
<00:25:13.590>agents? <00:25:13.745>What <00:25:13.901>are <00:25:14.056>the <00:25:14.212>biggest <00:25:14.367>ones <00:25:14.523>that <00:25:14.678>you <00:25:14.834>keep <00:25:14.990>hearing

432
00:25:17.510 --> 00:25:19.330
<00:25:17.510>about <00:25:17.710>that <00:25:17.910>they <00:25:18.110>want <00:25:18.310>an <00:25:18.510>agent <00:25:18.710>for <00:25:18.910>X?

433
00:25:19.810 --> 00:25:21.150
<00:25:19.810>One <00:25:20.111>thing <00:25:20.413>which <00:25:20.714>we <00:25:21.016>noticed,

434
00:25:22.510 --> 00:25:26.010
<00:25:22.510>and <00:25:22.626>of <00:25:22.743>course, <00:25:22.860>I'm <00:25:22.976>not <00:25:23.093>aware <00:25:23.210>of <00:25:23.326>all <00:25:23.443>the <00:25:23.560>things <00:25:23.676>that <00:25:23.793>we're <00:25:23.910>building.

435
00:25:26.110 --> 00:25:28.350
<00:25:26.110>I <00:25:26.250>can <00:25:26.389>speak <00:25:26.530>for <00:25:26.669>what <00:25:26.810>I'm <00:25:26.949>aware <00:25:27.090>of <00:25:27.229>at <00:25:27.370>the <00:25:27.510>moment.

436
00:25:28.950 --> 00:25:32.650
<00:25:28.950>A <00:25:29.125>lot <00:25:29.300>of <00:25:29.475>things <00:25:29.650>are <00:25:29.825>related <00:25:30.000>to <00:25:30.175>more <00:25:30.350>intelligent

437
00:25:32.730 --> 00:25:35.410
<00:25:32.730>search <00:25:33.010>across <00:25:33.290>huge <00:25:33.570>amount <00:25:33.850>of <00:25:34.130>data,

438
00:25:36.690 --> 00:25:40.450
<00:25:36.690>where <00:25:36.923>sometimes <00:25:37.156>even <00:25:37.390>humans, <00:25:37.623>even <00:25:37.856>subject <00:25:38.090>matter

439
00:25:40.490 --> 00:25:44.010
<00:25:40.490>experts <00:25:40.665>struggle, <00:25:40.840>but <00:25:41.015>you <00:25:41.190>can <00:25:41.365>actually <00:25:41.540>automate <00:25:41.715>this <00:25:41.890>or

440
00:25:44.290 --> 00:25:48.070
<00:25:44.290>partially <00:25:44.445>automate <00:25:44.601>this. <00:25:44.756>So <00:25:44.912>this <00:25:45.067>is <00:25:45.223>one <00:25:45.378>very <00:25:45.534>interesting <00:25:45.690>use

441
00:25:48.170 --> 00:25:51.770
<00:25:48.170>case. <00:25:48.370>And <00:25:48.570>then <00:25:48.770>other <00:25:48.970>interesting <00:25:49.170>use <00:25:49.370>cases <00:25:49.570>are

442
00:25:52.210 --> 00:25:55.380
<00:25:52.210>basically <00:25:52.676>automating <00:25:53.143>workflows, <00:25:53.610>right?

443
00:25:55.870 --> 00:25:59.490
<00:25:55.870>So <00:25:56.070>every, <00:25:56.270>let's <00:25:56.469>say, <00:25:56.669>human <00:25:56.870>workflow, <00:25:57.070>if <00:25:57.270>it's

444
00:25:59.510 --> 00:26:03.370
<00:25:59.510>reproducible <00:25:59.637>to <00:25:59.764>some <00:25:59.891>extent, <00:26:00.019>if <00:26:00.146>you <00:26:00.273>can <00:26:00.400>describe <00:26:00.528>it <00:26:00.655>in <00:26:00.782>natural <00:26:00.910>language,

445
00:26:04.750 --> 00:26:07.830
<00:26:04.750>it's <00:26:04.866>very <00:26:04.983>interesting <00:26:05.100>because <00:26:05.216>you <00:26:05.333>describe <00:26:05.450>it <00:26:05.566>in <00:26:05.683>natural <00:26:05.800>language, <00:26:05.916>you <00:26:06.033>give <00:26:06.150>the

446
00:26:07.870 --> 00:26:11.669
<00:26:07.870>right <00:26:08.025>tools <00:26:08.181>to <00:26:08.336>the <00:26:08.492>agent, <00:26:08.647>and <00:26:08.803>you <00:26:08.958>might <00:26:09.114>get <00:26:09.270>something

447
00:26:11.710 --> 00:26:13.600
<00:26:11.710>which <00:26:11.990>actually <00:26:12.270>kind <00:26:12.550>of <00:26:12.830>works, <00:26:13.110>right?

448
00:26:13.630 --> 00:26:17.470
<00:26:13.630>Then, <00:26:13.746>of <00:26:13.863>course, <00:26:13.980>you <00:26:14.096>still <00:26:14.213>need <00:26:14.330>the <00:26:14.446>human <00:26:14.563>in <00:26:14.680>the <00:26:14.796>loop <00:26:14.913>in <00:26:15.030>every

449
00:26:17.510 --> 00:26:20.830
<00:26:17.510>important <00:26:17.665>step, <00:26:17.821>right? <00:26:17.976>You <00:26:18.132>don't <00:26:18.287>want <00:26:18.443>your <00:26:18.598>agent <00:26:18.754>to <00:26:18.910>spend

450
00:26:20.850 --> 00:26:24.770
<00:26:20.850>$1,000 <00:26:21.129>on <00:26:21.409>something <00:26:21.689>without <00:26:21.969>your <00:26:22.250>actual

451
00:26:24.810 --> 00:26:28.030
<00:26:24.810>approval. <00:26:24.965>But <00:26:25.121>a <00:26:25.276>lot <00:26:25.432>of <00:26:25.587>those <00:26:25.743>workflows <00:26:25.898>can <00:26:26.054>be <00:26:26.210>automated.

452
00:26:28.910 --> 00:26:30.510
<00:26:28.910>And <00:26:29.190>also, <00:26:29.470>as <00:26:29.750>a <00:26:30.030>software <00:26:30.310>engineer,

453
00:26:31.670 --> 00:26:34.210
<00:26:31.670>we <00:26:31.845>are <00:26:32.020>doing <00:26:32.195>that <00:26:32.370>with <00:26:32.545>our <00:26:32.720>code <00:26:32.895>bases, <00:26:33.070>right?

454
00:26:34.290 --> 00:26:36.170
<00:26:34.290>So, <00:26:34.570>for <00:26:34.850>example, <00:26:35.129>when <00:26:35.409>we <00:26:35.690>are

455
00:26:37.010 --> 00:26:40.910
<00:26:37.010>introducing <00:26:37.210>something, <00:26:37.410>when <00:26:37.610>we <00:26:37.810>are <00:26:38.010>doing <00:26:38.210>some <00:26:38.410>development

456
00:26:40.990 --> 00:26:44.910
<00:26:40.990>workflow, <00:26:41.145>we <00:26:41.301>always <00:26:41.456>try <00:26:41.612>these <00:26:41.767>days <00:26:41.923>to <00:26:42.078>build <00:26:42.234>a <00:26:42.390>specific

457
00:26:44.970 --> 00:26:48.690
<00:26:44.970>skill <00:26:45.110>so <00:26:45.250>that <00:26:45.390>next <00:26:45.530>time <00:26:45.670>I <00:26:45.810>don't <00:26:45.950>have <00:26:46.090>to <00:26:46.230>redo <00:26:46.370>the

458
00:26:48.730 --> 00:26:52.510
<00:26:48.730>workflow <00:26:48.846>from <00:26:48.963>scratch. <00:26:49.080>I <00:26:49.196>can <00:26:49.313>just <00:26:49.430>prompt <00:26:49.546>Claude <00:26:49.663>and <00:26:49.780>say, <00:26:49.896>"Hey, <00:26:50.013>use <00:26:50.130>this

459
00:26:52.550 --> 00:26:53.870
<00:26:52.550>skill. <00:26:53.144>Do <00:26:53.737>whatever."

460
00:26:54.930 --> 00:26:58.750
<00:26:54.930>And <00:26:55.057>we <00:26:55.184>are <00:26:55.311>doing <00:26:55.439>that <00:26:55.566>for <00:26:55.693>things <00:26:55.820>like, <00:26:55.948>I <00:26:56.075>don't <00:26:56.202>know, <00:26:56.330>OWASP

461
00:26:58.770 --> 00:27:02.590
<00:26:58.770>security <00:26:58.970>scanning. <00:26:59.170>We <00:26:59.370>are <00:26:59.570>doing <00:26:59.770>that <00:26:59.970>for <00:27:00.170>traces

462
00:27:02.630 --> 00:27:06.550
<00:27:02.630>annotations, <00:27:02.746>so <00:27:02.863>that <00:27:02.980>feedback <00:27:03.096>loop <00:27:03.213>that <00:27:03.330>I <00:27:03.446>was <00:27:03.563>mentioning, <00:27:03.680>we <00:27:03.796>are <00:27:03.913>using <00:27:04.030>a

463
00:27:06.610 --> 00:27:10.020
<00:27:06.610>skill <00:27:06.717>to <00:27:06.825>do <00:27:06.933>that. <00:27:07.040>And <00:27:07.148>there <00:27:07.256>are <00:27:07.363>a <00:27:07.471>lot <00:27:07.579>of <00:27:07.686>things <00:27:07.794>that <00:27:07.902>we <00:27:08.010>can

464
00:27:10.790 --> 00:27:13.150
<00:27:10.790>automate <00:27:10.945>with, <00:27:11.101>of <00:27:11.256>course, <00:27:11.412>again, <00:27:11.567>the <00:27:11.723>human <00:27:11.878>in <00:27:12.034>the <00:27:12.190>loop.

465
00:27:13.770 --> 00:27:16.210
<00:27:13.770>Well, <00:27:13.925>that's <00:27:14.081>a <00:27:14.236>lot <00:27:14.392>of <00:27:14.547>stuff <00:27:14.703>that <00:27:14.858>people <00:27:15.014>can <00:27:15.170>start

466
00:27:17.490 --> 00:27:21.250
<00:27:17.490>getting <00:27:17.590>into. <00:27:17.690>What <00:27:17.790>would <00:27:17.890>you <00:27:17.990>say <00:27:18.090>is <00:27:18.190>a <00:27:18.290>good <00:27:18.390>starting <00:27:18.490>point <00:27:18.590>for <00:27:18.690>people <00:27:18.790>to <00:27:18.890>just--

467
00:27:21.370 --> 00:27:24.670
<00:27:21.370>We <00:27:21.525>talked <00:27:21.681>earlier <00:27:21.836>about <00:27:21.992>things <00:27:22.147>where <00:27:22.303>you're <00:27:22.458>inspired, <00:27:22.614>but <00:27:22.770>just

468
00:27:25.250 --> 00:27:27.770
<00:27:25.250>are <00:27:25.483>there <00:27:25.716>courses <00:27:25.950>that <00:27:26.183>you've <00:27:26.416>seen <00:27:26.650>online?

469
00:27:27.830 --> 00:27:30.470
<00:27:27.830>Are <00:27:27.946>there <00:27:28.063>things <00:27:28.179>that <00:27:28.296>you <00:27:28.413>provide <00:27:28.530>for <00:27:28.646>people <00:27:28.763>to <00:27:28.879>take <00:27:28.996>a <00:27:29.113>look <00:27:29.230>at?

470
00:27:30.810 --> 00:27:34.710
<00:27:30.810>Yeah, <00:27:31.010>actually. <00:27:31.210>So <00:27:31.409>I'm <00:27:31.610>personally <00:27:31.810>writing <00:27:32.010>a <00:27:32.210>book

471
00:27:34.750 --> 00:27:36.950
<00:27:34.750>with <00:27:34.950>O'Reilly <00:27:35.150>about <00:27:35.350>this. <00:27:35.550>It's <00:27:35.750>not <00:27:35.950>going <00:27:36.150>to

472
00:27:38.190 --> 00:27:40.270
<00:27:38.190>be <00:27:38.470>public <00:27:38.750>for <00:27:39.030>a <00:27:39.310>few <00:27:39.590>months.

473
00:27:41.330 --> 00:27:45.270
<00:27:41.330>But <00:27:41.470>we <00:27:41.610>are <00:27:41.750>planning <00:27:41.889>to <00:27:42.030>publish <00:27:42.169>the <00:27:42.310>early <00:27:42.449>release <00:27:42.590>soon, <00:27:42.730>which

474
00:27:45.310 --> 00:27:48.970
<00:27:45.310>is <00:27:45.485>called <00:27:45.659>"AI <00:27:45.835>Native <00:27:46.010>Software <00:27:46.185>Engineering." <00:27:46.360>So <00:27:46.534>that's <00:27:46.710>one

475
00:27:49.090 --> 00:27:52.470
<00:27:49.090>resources <00:27:49.206>which <00:27:49.323>I <00:27:49.439>hope <00:27:49.556>it's <00:27:49.673>going <00:27:49.790>to <00:27:49.906>be <00:27:50.023>interesting <00:27:50.139>soon <00:27:50.256>in <00:27:50.373>the <00:27:50.490>future.

476
00:27:53.050 --> 00:27:56.770
<00:27:53.050>But <00:27:53.132>if <00:27:53.214>you <00:27:53.297>had <00:27:53.379>to <00:27:53.461>start <00:27:53.544>now, <00:27:53.626>I <00:27:53.708>would <00:27:53.791>say <00:27:53.873>that <00:27:53.955>you <00:27:54.038>can <00:27:54.120>go <00:27:54.202>on <00:27:54.285>things <00:27:54.367>like <00:27:54.450>the

477
00:27:56.810 --> 00:28:00.270
<00:27:56.810>Anthropic <00:27:56.937>blog. <00:27:57.064>Anthropic <00:27:57.191>has <00:27:57.319>a <00:27:57.446>lot <00:27:57.573>of <00:27:57.700>great <00:27:57.828>courses, <00:27:57.955>to <00:27:58.082>be <00:27:58.210>fair.

478
00:28:01.090 --> 00:28:04.710
<00:28:01.090>So <00:28:01.230>that <00:28:01.370>is <00:28:01.510>one. <00:28:01.649>And <00:28:01.790>then <00:28:01.929>there <00:28:02.070>is <00:28:02.209>the <00:28:02.350>deep <00:28:02.490>learning

479
00:28:05.290 --> 00:28:08.130
<00:28:05.290>platform <00:28:05.490>from <00:28:05.690>Andrew <00:28:05.889>Ng, <00:28:06.090>which <00:28:06.290>is <00:28:06.490>just <00:28:06.690>amazing.

480
00:28:08.310 --> 00:28:09.890
<00:28:08.310>So <00:28:08.485>I <00:28:08.659>would <00:28:08.835>recommend <00:28:09.010>looking <00:28:09.185>at <00:28:09.360>that <00:28:09.534>as <00:28:09.710>well.

481
00:28:10.370 --> 00:28:13.590
<00:28:10.370>So <00:28:10.570>you <00:28:10.770>sent <00:28:10.969>us <00:28:11.169>a <00:28:11.370>ton <00:28:11.570>of <00:28:11.770>talk

482
00:28:13.670 --> 00:28:17.270
<00:28:13.670>proposals. <00:28:13.810>So <00:28:13.950>what <00:28:14.090>are <00:28:14.230>you <00:28:14.370>most <00:28:14.510>excited <00:28:14.650>about <00:28:14.790>the <00:28:14.930>World <00:28:15.070>Congress?

483
00:28:17.310 --> 00:28:21.130
<00:28:17.310>Have <00:28:17.397>you <00:28:17.485>been <00:28:17.572>at <00:28:17.659>it <00:28:17.747>already, <00:28:17.835>or <00:28:17.922>are <00:28:18.010>you <00:28:18.097>looking <00:28:18.185>forward <00:28:18.272>to <00:28:18.360>it, <00:28:18.447>and <00:28:18.534>what <00:28:18.622>are <00:28:18.710>you

484
00:28:21.170 --> 00:28:23.390
<00:28:21.170>most <00:28:21.345>excited <00:28:21.520>about <00:28:21.695>when <00:28:21.870>it <00:28:22.045>comes <00:28:22.220>to <00:28:22.395>the <00:28:22.570>Congress?

485
00:28:23.870 --> 00:28:27.210
<00:28:23.870>Yeah, <00:28:23.986>I've <00:28:24.103>been <00:28:24.219>there <00:28:24.336>in, <00:28:24.453>I <00:28:24.570>think <00:28:24.686>it <00:28:24.803>was <00:28:24.919>2024 <00:28:25.036>as <00:28:25.153>a <00:28:25.270>speaker,

486
00:28:27.870 --> 00:28:31.570
<00:28:27.870>and <00:28:28.025>I've <00:28:28.181>loved <00:28:28.336>my <00:28:28.492>time <00:28:28.647>there. <00:28:28.803>I've <00:28:28.958>met <00:28:29.114>so <00:28:29.270>many

487
00:28:31.630 --> 00:28:35.470
<00:28:31.630>incredible <00:28:31.805>people. <00:28:31.980>So <00:28:32.155>yeah, <00:28:32.330>I <00:28:32.505>think <00:28:32.680>that's <00:28:32.855>the <00:28:33.030>thing.

488
00:28:36.230 --> 00:28:40.070
<00:28:36.230>When <00:28:36.346>you <00:28:36.463>are <00:28:36.580>in <00:28:36.696>a <00:28:36.813>conference <00:28:36.930>like <00:28:37.046>the <00:28:37.163>World <00:28:37.280>Congress, <00:28:37.396>you <00:28:37.513>can <00:28:37.630>learn

489
00:28:40.130 --> 00:28:43.190
<00:28:40.130>quite <00:28:40.285>a <00:28:40.441>lot, <00:28:40.596>and <00:28:40.752>you <00:28:40.907>can <00:28:41.063>meet <00:28:41.218>so <00:28:41.374>many <00:28:41.530>people.

490
00:28:43.650 --> 00:28:46.910
<00:28:43.650>Just <00:28:43.825>connecting <00:28:44.000>with <00:28:44.175>others, <00:28:44.350>I <00:28:44.525>think <00:28:44.700>it's <00:28:44.875>a <00:28:45.050>great

491
00:28:46.930 --> 00:28:50.470
<00:28:46.930>experience. <00:28:47.105>And <00:28:47.280>I <00:28:47.455>think <00:28:47.630>in <00:28:47.805>those <00:28:47.980>couple <00:28:48.155>of <00:28:48.330>days,

492
00:28:51.490 --> 00:28:54.130
<00:28:51.490>I've <00:28:51.690>learned <00:28:51.890>so <00:28:52.090>much <00:28:52.290>from <00:28:52.490>so <00:28:52.690>many <00:28:52.890>people.

493
00:28:54.470 --> 00:28:57.810
<00:28:54.470>I <00:28:54.597>was <00:28:54.724>just <00:28:54.851>going <00:28:54.979>around <00:28:55.106>looking <00:28:55.233>at <00:28:55.360>talks, <00:28:55.488>and <00:28:55.615>it <00:28:55.742>was <00:28:55.870>basically

494
00:28:57.870 --> 00:29:01.690
<00:28:57.870>impossible <00:28:58.070>for <00:28:58.270>me <00:28:58.469>to <00:28:58.669>follow <00:28:58.870>all <00:28:59.070>the <00:28:59.270>interesting

495
00:29:01.790 --> 00:29:04.830
<00:29:01.790>talks, <00:29:01.945>because <00:29:02.101>a <00:29:02.256>lot <00:29:02.412>of <00:29:02.567>tracks <00:29:02.723>were <00:29:02.878>in <00:29:03.034>parallel, <00:29:03.190>right?

496
00:29:06.290 --> 00:29:09.190
<00:29:06.290>So <00:29:06.490>yeah, <00:29:06.690>that's <00:29:06.889>what <00:29:07.090>excites <00:29:07.290>me, <00:29:07.490>is <00:29:07.690>making

497
00:29:09.230 --> 00:29:12.870
<00:29:09.230>me <00:29:09.510>more <00:29:09.790>excited, <00:29:10.070>just <00:29:10.350>meeting <00:29:10.630>incredible

498
00:29:12.970 --> 00:29:16.190
<00:29:12.970>people, <00:29:13.320>having <00:29:13.670>very <00:29:14.020>interesting <00:29:14.370>conversations,

499
00:29:16.870 --> 00:29:19.920
<00:29:16.870>and <00:29:17.010>yep, <00:29:17.149>I <00:29:17.290>hope <00:29:17.429>to <00:29:17.570>be <00:29:17.709>there <00:29:17.850>this <00:29:17.989>year <00:29:18.129>as <00:29:18.270>well.

500
00:29:20.119 --> 00:29:23.469
<00:29:20.119>I <00:29:20.259>always <00:29:20.399>was <00:29:20.539>fascinated <00:29:20.678>that <00:29:20.819>you <00:29:20.958>meet <00:29:21.099>so <00:29:21.238>many <00:29:21.379>technical <00:29:21.519>people.

501
00:29:23.529 --> 00:29:27.290
<00:29:23.529>It's <00:29:23.622>like <00:29:23.715>at <00:29:23.809>a <00:29:23.902>lot <00:29:23.995>of <00:29:24.089>conferences, <00:29:24.182>you <00:29:24.275>either <00:29:24.369>meet <00:29:24.462>the <00:29:24.555>people <00:29:24.649>you <00:29:24.742>already <00:29:24.835>know <00:29:24.929>if

502
00:29:27.330 --> 00:29:31.300
<00:29:27.330>you're <00:29:27.417>a <00:29:27.504>speaker <00:29:27.592>and <00:29:27.679>if <00:29:27.767>you've <00:29:27.855>been <00:29:27.942>at <00:29:28.030>a <00:29:28.117>few <00:29:28.205>of <00:29:28.292>them, <00:29:28.379>or <00:29:28.467>you <00:29:28.554>meet <00:29:28.642>people <00:29:28.730>from

503
00:29:31.370 --> 00:29:33.510
<00:29:31.370>companies <00:29:31.544>that <00:29:31.719>you <00:29:31.895>can <00:29:32.070>talk <00:29:32.245>to <00:29:32.419>about <00:29:32.594>their <00:29:32.770>products.

504
00:29:33.830 --> 00:29:37.590
<00:29:33.830>But <00:29:33.957>even <00:29:34.084>in <00:29:34.211>the <00:29:34.339>audience <00:29:34.466>at <00:29:34.593>the <00:29:34.720>Developer <00:29:34.848>World <00:29:34.975>Congress, <00:29:35.102>I <00:29:35.230>always

505
00:29:37.630 --> 00:29:39.680
<00:29:37.630>found <00:29:37.980>people <00:29:38.330>are <00:29:38.680>all <00:29:39.030>technical.

506
00:29:39.760 --> 00:29:39.760
Right.

507
00:29:39.790 --> 00:29:40.030
There's

508
00:29:41.530 --> 00:29:44.990
<00:29:41.530>no <00:29:41.810>salespeople <00:29:42.090>that <00:29:42.370>actually <00:29:42.649>pester <00:29:42.930>you.

509
00:29:45.730 --> 00:29:48.510
<00:29:45.730>Not <00:29:45.905>anything <00:29:46.080>against <00:29:46.255>salespeople. <00:29:46.430>We <00:29:46.605>need <00:29:46.780>them <00:29:46.955>for <00:29:47.130>sure.

510
00:29:48.530 --> 00:29:52.490
<00:29:48.530>But <00:29:48.657>it's <00:29:48.784>incredible <00:29:48.911>how <00:29:49.039>we <00:29:49.166>managed <00:29:49.293>to <00:29:49.420>attract <00:29:49.548>that <00:29:49.675>many <00:29:49.802>technical <00:29:49.930>folk

511
00:29:52.590 --> 00:29:55.710
<00:29:52.590>rather <00:29:52.730>than <00:29:52.870>just <00:29:53.010>the <00:29:53.149>people <00:29:53.290>in <00:29:53.429>the <00:29:53.570>surrounding <00:29:53.709>of <00:29:53.850>technical <00:29:53.990>folk.

512
00:29:56.550 --> 00:30:00.010
<00:29:56.550>Yeah. <00:29:56.677>I <00:29:56.804>mean, <00:29:56.931>it's <00:29:57.059>one <00:29:57.186>of <00:29:57.313>those <00:29:57.440>conferences <00:29:57.568>where <00:29:57.695>you <00:29:57.822>know <00:29:57.950>that

513
00:30:01.530 --> 00:30:05.450
<00:30:01.530>both <00:30:01.637>as <00:30:01.745>a <00:30:01.853>speaker <00:30:01.960>and <00:30:02.068>as <00:30:02.176>an <00:30:02.283>attendee, <00:30:02.391>you <00:30:02.499>go <00:30:02.606>there <00:30:02.714>to <00:30:02.822>stay <00:30:02.930>with

514
00:30:05.470 --> 00:30:08.730
<00:30:05.470>technical <00:30:05.703>people, <00:30:05.936>enjoy <00:30:06.170>some <00:30:06.403>quality <00:30:06.636>time <00:30:06.870>together,

515
00:30:09.330 --> 00:30:09.980
<00:30:09.330>and <00:30:09.622>yeah, <00:30:09.915>I

516
00:30:12.070 --> 00:30:15.490
<00:30:12.070>think <00:30:12.225>you <00:30:12.381>summarized <00:30:12.536>it <00:30:12.692>correctly. <00:30:12.847>It's <00:30:13.003>not <00:30:13.158>like <00:30:13.314>sales <00:30:13.470>pitchy.

517
00:30:15.510 --> 00:30:17.370
<00:30:15.510>It's <00:30:15.790>not <00:30:16.070>like <00:30:16.350>marketing <00:30:16.629>and <00:30:16.910>stuff.

518
00:30:17.490 --> 00:30:21.170
<00:30:17.490>It's <00:30:17.690>people <00:30:17.890>building <00:30:18.090>cool <00:30:18.290>stuff, <00:30:18.490>talking <00:30:18.690>about <00:30:18.890>it,

519
00:30:21.590 --> 00:30:22.770
<00:30:21.590>and <00:30:21.855>having <00:30:22.120>fun. <00:30:22.386>That's <00:30:22.652>it.

520
00:30:23.550 --> 00:30:27.520
<00:30:23.550>Now, <00:30:23.649>if <00:30:23.750>anybody <00:30:23.850>is <00:30:23.950>still <00:30:24.050>on <00:30:24.149>the <00:30:24.250>fence <00:30:24.350>considering <00:30:24.450>to <00:30:24.550>go <00:30:24.649>to <00:30:24.750>the <00:30:24.850>Congress <00:30:24.950>and

521
00:30:27.590 --> 00:30:30.350
<00:30:27.590>ask <00:30:27.730>their <00:30:27.870>boss <00:30:28.010>for <00:30:28.149>a <00:30:28.290>ticket, <00:30:28.429>what <00:30:28.570>would <00:30:28.709>you <00:30:28.850>tell <00:30:28.990>them?

522
00:30:30.490 --> 00:30:33.110
<00:30:30.490>Why <00:30:30.645>should <00:30:30.801>you <00:30:30.956>go <00:30:31.112>to <00:30:31.267>the <00:30:31.423>Congress <00:30:31.578>as <00:30:31.734>an <00:30:31.890>attendee?

523
00:30:33.750 --> 00:30:37.490
<00:30:33.750>I <00:30:33.925>definitely <00:30:34.100>believe <00:30:34.275>that <00:30:34.450>joining <00:30:34.625>the <00:30:34.800>Congress <00:30:34.975>and <00:30:35.150>visiting

524
00:30:37.570 --> 00:30:40.470
<00:30:37.570>Berlin <00:30:37.803>in <00:30:38.036>general, <00:30:38.270>it's <00:30:38.503>a <00:30:38.736>great <00:30:38.970>opportunity.

525
00:30:41.790 --> 00:30:45.730
<00:30:41.790>I <00:30:41.889>had <00:30:41.990>a <00:30:42.090>lot <00:30:42.190>of <00:30:42.290>fun, <00:30:42.389>and <00:30:42.490>I <00:30:42.590>really <00:30:42.690>hope <00:30:42.790>to <00:30:42.889>be <00:30:42.990>joining <00:30:43.090>this <00:30:43.190>year.

526
00:30:46.810 --> 00:30:50.410
<00:30:46.810>It's <00:30:47.010>a <00:30:47.210>great <00:30:47.409>opportunity <00:30:47.610>for <00:30:47.810>your <00:30:48.010>career, <00:30:48.210>for

527
00:30:50.550 --> 00:30:54.450
<00:30:50.550>your <00:30:50.750>learning, <00:30:50.950>and <00:30:51.149>for, <00:30:51.350>again, <00:30:51.550>do <00:30:51.750>networking <00:30:51.950>with

528
00:30:54.510 --> 00:30:58.490
<00:30:54.510>very <00:30:54.626>interesting <00:30:54.743>people. <00:30:54.860>So <00:30:54.976>if <00:30:55.093>you <00:30:55.210>are <00:30:55.326>considering <00:30:55.443>to <00:30:55.560>join <00:30:55.676>the <00:30:55.793>World <00:30:55.910>Congress,

529
00:30:59.210 --> 00:31:02.410
<00:30:59.210>yeah, <00:30:59.350>that's <00:30:59.490>your <00:30:59.630>chance <00:30:59.770>to <00:30:59.910>do <00:31:00.050>so. <00:31:00.190>It's <00:31:00.330>definitely <00:31:00.470>worth <00:31:00.610>it.

530
00:31:02.970 --> 00:31:06.850
<00:31:02.970>Well, <00:31:03.110>thanks <00:31:03.250>very <00:31:03.390>much. <00:31:03.530>This <00:31:03.670>was <00:31:03.810>Alfonso <00:31:03.950>Graziano, <00:31:04.090>tech <00:31:04.230>lead <00:31:04.370>at

531
00:31:06.870 --> 00:31:08.350
<00:31:06.870>NearForm, <00:31:07.313>talking <00:31:07.757>about <00:31:08.202>agents.

532
00:31:09.230 --> 00:31:12.220
<00:31:09.230>We <00:31:09.323>might <00:31:09.416>have <00:31:09.510>you <00:31:09.603>back <00:31:09.696>on <00:31:09.790>the <00:31:09.883>show <00:31:09.976>because <00:31:10.070>you <00:31:10.163>sent <00:31:10.256>so <00:31:10.350>many <00:31:10.443>cool <00:31:10.536>things <00:31:10.630>in.

533
00:31:12.350 --> 00:31:16.030
<00:31:12.350>I <00:31:12.443>can't <00:31:12.536>promise <00:31:12.629>that <00:31:12.723>you <00:31:12.816>will <00:31:12.909>be <00:31:13.003>booked <00:31:13.096>as <00:31:13.189>a <00:31:13.283>speaker, <00:31:13.376>but <00:31:13.469>there <00:31:13.563>is <00:31:13.656>a <00:31:13.750>chance

534
00:31:16.110 --> 00:31:18.290
<00:31:16.110>that <00:31:16.310>we <00:31:16.510>will <00:31:16.709>see <00:31:16.909>you <00:31:17.110>on <00:31:17.310>stage <00:31:17.510>again.

535
00:31:18.330 --> 00:31:21.330
<00:31:18.330>And <00:31:18.407>if <00:31:18.485>not, <00:31:18.563>we're <00:31:18.641>going <00:31:18.718>to <00:31:18.796>see <00:31:18.874>you <00:31:18.952>on <00:31:19.030>more <00:31:19.107>of <00:31:19.185>these <00:31:19.263>shows <00:31:19.341>because <00:31:19.418>this <00:31:19.496>was <00:31:19.574>a <00:31:19.652>lot <00:31:19.730>of

536
00:31:21.410 --> 00:31:25.390
<00:31:21.410>fun. <00:31:21.503>Thanks <00:31:21.596>very <00:31:21.690>much. <00:31:21.783>Any <00:31:21.876>last <00:31:21.970>words <00:31:22.063>you <00:31:22.156>have <00:31:22.250>for <00:31:22.343>the <00:31:22.436>audience <00:31:22.530>out <00:31:22.623>there, <00:31:22.716>for <00:31:22.810>the

537
00:31:25.450 --> 00:31:25.950
internets?

538
00:31:26.870 --> 00:31:30.590
<00:31:26.870>No, <00:31:27.103>just <00:31:27.336>let's <00:31:27.570>building <00:31:27.803>agents <00:31:28.036>securely, <00:31:28.270>hopefully,

539
00:31:31.130 --> 00:31:33.430
<00:31:31.130>and <00:31:31.330>let's <00:31:31.530>have <00:31:31.730>fun <00:31:31.930>with <00:31:32.130>this <00:31:32.330>amazing <00:31:32.530>technology.

540
00:31:34.410 --> 00:31:37.750
<00:31:34.410>Thanks <00:31:34.550>very <00:31:34.690>much. <00:31:34.830>This <00:31:34.970>was <00:31:35.110>"Coffee <00:31:35.250>with <00:31:35.390>Developers" <00:31:35.530>with <00:31:35.670>Alfonso <00:31:35.810>Graziano.

541
00:31:37.810 --> 00:31:41.210
<00:31:37.810>I <00:31:37.917>just <00:31:38.025>love <00:31:38.133>saying <00:31:38.240>that <00:31:38.348>word, <00:31:38.456>that <00:31:38.563>name. <00:31:38.671>It <00:31:38.779>just <00:31:38.886>feels <00:31:38.994>like <00:31:39.102>a <00:31:39.210>song.

542
00:31:41.650 --> 00:31:44.310
<00:31:41.650>This <00:31:41.825>is <00:31:42.000>wonderful. <00:31:42.175>Thanks <00:31:42.350>again, <00:31:42.525>Alfonso. <00:31:42.700>This <00:31:42.875>was <00:31:43.050>wonderful.

543
00:31:44.930 --> 00:31:46.690
<00:31:44.930>Thanks <00:31:45.210>a <00:31:45.490>lot, <00:31:45.770>Chris. <00:31:46.050>Cheers, <00:31:46.330>everybody.