WEBVTT

1
00:00:01.300 --> 00:00:03.860
<00:00:01.300>Hello <00:00:01.533>everyone. <00:00:01.766>My <00:00:02.000>name <00:00:02.233>is <00:00:02.466>Akshay <00:00:02.700>Nagpal.

2
00:00:03.940 --> 00:00:05.500
<00:00:03.940>I'm <00:00:04.290>an <00:00:04.640>agentic <00:00:04.990>product <00:00:05.340>lead.

3
00:00:07.640 --> 00:00:11.520
<00:00:07.640>Today <00:00:07.840>at <00:00:08.039>VR <00:00:08.240>Developers <00:00:08.440>2026, <00:00:08.639>we <00:00:08.840>are <00:00:09.040>going

4
00:00:11.560 --> 00:00:15.500
<00:00:11.560>to <00:00:11.715>talk <00:00:11.871>about <00:00:12.026>why <00:00:12.182>most <00:00:12.337>AI <00:00:12.493>features <00:00:12.648>fail <00:00:12.804>after <00:00:12.960>the

5
00:00:15.540 --> 00:00:15.880
demo.

6
00:00:16.720 --> 00:00:20.599
<00:00:16.720>And <00:00:16.860>we're <00:00:17.000>going <00:00:17.140>to <00:00:17.279>see <00:00:17.419>and <00:00:17.560>think <00:00:17.700>about <00:00:17.840>how <00:00:17.979>we <00:00:18.119>can

7
00:00:20.670 --> 00:00:24.520
<00:00:20.670>create <00:00:20.903>AI <00:00:21.136>systems <00:00:21.370>that <00:00:21.603>people <00:00:21.836>can <00:00:22.070>actually

8
00:00:24.560 --> 00:00:27.940
<00:00:24.560>trust, <00:00:24.840>adopt, <00:00:25.119>and <00:00:25.400>use <00:00:25.680>in <00:00:25.959>production.

9
00:00:27.980 --> 00:00:31.560
<00:00:27.980>So <00:00:28.135>we're <00:00:28.291>going <00:00:28.446>to <00:00:28.602>talk <00:00:28.757>about <00:00:28.913>the <00:00:29.068>seams <00:00:29.224>that <00:00:29.380>we

10
00:00:31.600 --> 00:00:35.340
<00:00:31.600>create, <00:00:31.740>the <00:00:31.880>handoffs <00:00:32.020>that <00:00:32.160>we <00:00:32.300>create <00:00:32.439>as <00:00:32.580>part <00:00:32.720>of <00:00:32.860>the <00:00:33.000>AI

11
00:00:35.380 --> 00:00:37.340
<00:00:35.380>systems <00:00:35.660>and <00:00:35.940>features <00:00:36.220>that <00:00:36.500>we <00:00:36.780>build.

12
00:00:39.060 --> 00:00:42.320
<00:00:39.060>So <00:00:39.340>we're <00:00:39.620>going <00:00:39.900>to <00:00:40.180>start <00:00:40.460>with

13
00:00:44.800 --> 00:00:48.750
<00:00:44.800>the <00:00:44.916>problems. <00:00:45.033>Number <00:00:45.150>one, <00:00:45.266>what <00:00:45.383>are <00:00:45.500>the <00:00:45.616>problems <00:00:45.733>that <00:00:45.849>we <00:00:45.966>see <00:00:46.083>in <00:00:46.199>production?

14
00:00:49.340 --> 00:00:52.720
<00:00:49.340>Number <00:00:49.515>two, <00:00:49.690>the <00:00:49.865>five <00:00:50.040>failure <00:00:50.215>modes <00:00:50.390>that <00:00:50.565>are <00:00:50.740>there

15
00:00:53.460 --> 00:00:57.420
<00:00:53.460>that <00:00:53.615>are <00:00:53.771>very <00:00:53.926>common <00:00:54.082>out <00:00:54.237>there. <00:00:54.393>Number <00:00:54.548>three, <00:00:54.704>a <00:00:54.860>framework

16
00:00:57.460 --> 00:01:00.260
<00:00:57.460>on <00:00:57.660>how <00:00:57.860>to <00:00:58.060>build <00:00:58.260>AI <00:00:58.460>products <00:00:58.660>for <00:00:58.860>trust.

17
00:01:00.820 --> 00:01:04.761
<00:01:00.820>Number <00:01:00.947>four, <00:01:01.074>we're <00:01:01.201>going <00:01:01.329>to <00:01:01.456>see <00:01:01.583>some <00:01:01.710>examples <00:01:01.838>of <00:01:01.965>how <00:01:02.092>to <00:01:02.220>do

18
00:01:04.840 --> 00:01:07.260
<00:01:04.840>it <00:01:05.190>in <00:01:05.540>practice. <00:01:05.890>Number <00:01:06.240>five,

19
00:01:08.680 --> 00:01:12.520
<00:01:08.680>I'm <00:01:08.807>going <00:01:08.934>to <00:01:09.061>give <00:01:09.189>you <00:01:09.316>a <00:01:09.443>checklist <00:01:09.570>that <00:01:09.698>you <00:01:09.825>can <00:01:09.952>take <00:01:10.080>back

20
00:01:13.580 --> 00:01:16.550
<00:01:13.580>whenever <00:01:13.707>you <00:01:13.834>build <00:01:13.961>a <00:01:14.089>new <00:01:14.216>AI <00:01:14.343>feature <00:01:14.470>that <00:01:14.598>you <00:01:14.725>can <00:01:14.852>run <00:01:14.980>it

21
00:01:16.580 --> 00:01:20.300
<00:01:16.580>by. <00:01:16.860>So <00:01:17.140>coming <00:01:17.420>to <00:01:17.700>the <00:01:17.980>problem,

22
00:01:21.080 --> 00:01:24.880
<00:01:21.080>why <00:01:21.280>is <00:01:21.480>it <00:01:21.679>that <00:01:21.880>all <00:01:22.080>the <00:01:22.280>AI <00:01:22.480>systems,

23
00:01:24.960 --> 00:01:26.940
<00:01:24.960>they <00:01:25.426>look <00:01:25.893>amazingly <00:01:26.360>well

24
00:01:27.800 --> 00:01:30.940
<00:01:27.800>when <00:01:27.975>we <00:01:28.149>are <00:01:28.325>demoing <00:01:28.500>them, <00:01:28.675>but <00:01:28.850>they <00:01:29.024>quietly <00:01:29.200>die

25
00:01:31.580 --> 00:01:35.220
<00:01:31.580>when <00:01:31.707>we <00:01:31.834>release <00:01:31.961>them <00:01:32.089>to <00:01:32.216>production <00:01:32.343>and <00:01:32.470>give <00:01:32.598>them <00:01:32.725>out <00:01:32.852>to <00:01:32.980>our

26
00:01:35.280 --> 00:01:35.760
users?

27
00:01:38.120 --> 00:01:38.420
And

28
00:01:39.960 --> 00:01:43.420
<00:01:39.960>we <00:01:40.193>all <00:01:40.426>know <00:01:40.660>that <00:01:40.893>every <00:01:41.126>AI <00:01:41.360>feature

29
00:01:44.500 --> 00:01:47.620
<00:01:44.500>starts <00:01:44.655>as <00:01:44.811>a <00:01:44.966>hero. <00:01:45.122>In <00:01:45.277>a <00:01:45.433>demo, <00:01:45.588>everything <00:01:45.744>is <00:01:45.900>perfect.

30
00:01:50.560 --> 00:01:54.480
<00:01:50.560>We <00:01:50.760>give <00:01:50.960>curated <00:01:51.160>inputs. <00:01:51.360>We <00:01:51.560>have <00:01:51.760>cherry-picked <00:01:51.960>prompts

31
00:01:54.540 --> 00:01:58.010
<00:01:54.539>that <00:01:54.679>we <00:01:54.820>give. <00:01:54.960>The <00:01:55.100>audience <00:01:55.240>is <00:01:55.380>really <00:01:55.520>excited <00:01:55.660>to <00:01:55.800>see <00:01:55.940>this

32
00:01:58.040 --> 00:02:01.570
<00:01:58.039>novelty. <00:01:58.240>We <00:01:58.440>generally <00:01:58.639>only <00:01:58.839>try <00:01:59.039>the <00:01:59.240>happy <00:01:59.440>paths,

33
00:02:03.120 --> 00:02:06.440
<00:02:03.120>and <00:02:03.470>99% <00:02:03.820>of <00:02:04.170>the <00:02:04.520>magic

34
00:02:08.021 --> 00:02:11.180
<00:02:08.021>lives <00:02:08.148>in <00:02:08.275>the <00:02:08.402>1% <00:02:08.530>of <00:02:08.657>the <00:02:08.784>cases <00:02:08.911>that <00:02:09.039>we <00:02:09.166>try <00:02:09.293>to <00:02:09.421>show,

35
00:02:11.880 --> 00:02:15.000
<00:02:11.880>whereas <00:02:12.230>the <00:02:12.579>99% <00:02:12.930>of <00:02:13.280>cases,

36
00:02:16.280 --> 00:02:19.800
<00:02:16.280>where <00:02:16.455>it <00:02:16.630>fails <00:02:16.805>is <00:02:16.980>where <00:02:17.155>user <00:02:17.330>actually <00:02:17.505>lives <00:02:17.680>or

37
00:02:19.840 --> 00:02:20.350
tries.

38
00:02:21.260 --> 00:02:21.340
So

39
00:02:23.500 --> 00:02:25.700
<00:02:23.500>this <00:02:23.850>is <00:02:24.200>how <00:02:24.550>we <00:02:24.900>expect

40
00:02:27.140 --> 00:02:30.940
<00:02:27.140>how <00:02:27.315>the <00:02:27.489>adoption <00:02:27.665>would <00:02:27.839>grow. <00:02:28.015>We <00:02:28.190>think <00:02:28.364>of <00:02:28.540>it

41
00:02:31.000 --> 00:02:34.800
<00:02:31.000>like <00:02:31.107>the <00:02:31.215>blue <00:02:31.323>line <00:02:31.430>on <00:02:31.538>the <00:02:31.646>graph, <00:02:31.753>that <00:02:31.861>it <00:02:31.969>would <00:02:32.076>be <00:02:32.184>like <00:02:32.292>a <00:02:32.400>hockey

42
00:02:34.900 --> 00:02:35.280
stick.

43
00:02:36.400 --> 00:02:40.180
<00:02:36.400>We <00:02:36.527>are <00:02:36.654>going <00:02:36.781>to <00:02:36.909>launch <00:02:37.036>and <00:02:37.163>slowly <00:02:37.290>and <00:02:37.418>surely <00:02:37.545>it's <00:02:37.672>going <00:02:37.800>to

44
00:02:40.240 --> 00:02:43.660
<00:02:40.240>rise, <00:02:40.440>the <00:02:40.640>adoption, <00:02:40.840>and <00:02:41.040>it's <00:02:41.240>going <00:02:41.440>to <00:02:41.640>continue.

45
00:02:43.730 --> 00:02:46.920
<00:02:43.730>So <00:02:43.885>we <00:02:44.041>assume <00:02:44.196>that <00:02:44.352>there <00:02:44.507>is <00:02:44.663>going <00:02:44.818>to <00:02:44.974>be <00:02:45.130>uptake,

46
00:02:47.660 --> 00:02:50.160
<00:02:47.660>word <00:02:47.815>of <00:02:47.971>mouth, <00:02:48.126>there's <00:02:48.282>going <00:02:48.437>to <00:02:48.593>be <00:02:48.748>retention <00:02:48.904>of <00:02:49.060>the

47
00:02:50.200 --> 00:02:54.120
<00:02:50.200>product. <00:02:50.433>However, <00:02:50.666>what <00:02:50.899>we <00:02:51.133>generally <00:02:51.366>see <00:02:51.600>is

48
00:02:54.180 --> 00:02:57.780
<00:02:54.180>in <00:02:54.307>the <00:02:54.434>beginning, <00:02:54.561>because <00:02:54.689>of <00:02:54.816>the <00:02:54.943>novelty, <00:02:55.070>we <00:02:55.198>see <00:02:55.325>a <00:02:55.452>spike, <00:02:55.580>and

49
00:02:57.840 --> 00:02:59.680
<00:02:57.840>then <00:02:58.190>we <00:02:58.540>start <00:02:58.890>seeing <00:02:59.240>churn,

50
00:03:00.500 --> 00:03:02.880
<00:03:00.500>and <00:03:00.733>the <00:03:00.966>feature <00:03:01.200>goes <00:03:01.433>to <00:03:01.666>the <00:03:01.900>graveyard.

51
00:03:03.750 --> 00:03:07.440
<00:03:03.750>We <00:03:04.030>see <00:03:04.310>massive <00:03:04.590>churn, <00:03:04.870>low <00:03:05.150>retention,

52
00:03:09.200 --> 00:03:12.600
<00:03:09.200>no <00:03:09.399>more <00:03:09.600>uptake <00:03:09.799>once <00:03:10.000>the <00:03:10.200>initial <00:03:10.399>excitement <00:03:10.600>has

53
00:03:13.940 --> 00:03:14.820
<00:03:13.940>kind <00:03:14.204>of <00:03:14.468>gone <00:03:14.732>down.

54
00:03:17.580 --> 00:03:20.600
<00:03:17.579>So <00:03:17.719>why <00:03:17.859>is <00:03:17.999>it <00:03:18.140>that <00:03:18.279>the <00:03:18.420>demo <00:03:18.559>actually <00:03:18.700>lied <00:03:18.839>to <00:03:18.980>us?

55
00:03:20.880 --> 00:03:24.200
<00:03:20.880>Why <00:03:21.035>is <00:03:21.191>everything <00:03:21.346>working <00:03:21.502>in <00:03:21.657>demo <00:03:21.813>whereas <00:03:21.968>not <00:03:22.124>in <00:03:22.280>production?

56
00:03:25.980 --> 00:03:26.420
<00:03:25.980>Number <00:03:26.376>one,

57
00:03:28.220 --> 00:03:32.180
<00:03:28.220>demo <00:03:28.375>is <00:03:28.531>actually <00:03:28.686>engineered <00:03:28.842>to <00:03:28.997>succeed, <00:03:29.153>and <00:03:29.308>the <00:03:29.464>builder <00:03:29.620>of

58
00:03:32.220 --> 00:03:34.420
<00:03:32.220>the <00:03:32.420>product <00:03:32.620>is <00:03:32.820>generally <00:03:33.020>the <00:03:33.220>one <00:03:33.420>driving <00:03:33.620>it.

59
00:03:35.500 --> 00:03:38.760
<00:03:35.500>They <00:03:35.607>know <00:03:35.715>exactly <00:03:35.823>how <00:03:35.930>the <00:03:36.038>system <00:03:36.146>works, <00:03:36.253>so <00:03:36.361>they <00:03:36.469>know <00:03:36.576>exactly <00:03:36.684>what <00:03:36.792>to <00:03:36.900>type.

60
00:03:39.340 --> 00:03:42.900
<00:03:39.340>Every <00:03:39.480>input <00:03:39.620>is <00:03:39.760>pristine, <00:03:39.900>so <00:03:40.040>there's <00:03:40.180>no <00:03:40.320>typo, <00:03:40.460>no <00:03:40.600>ambiguity, <00:03:40.740>no

61
00:03:42.960 --> 00:03:46.710
<00:03:42.960>adversarial <00:03:43.160>user, <00:03:43.360>whereas <00:03:43.560>production <00:03:43.760>is <00:03:43.960>opposite <00:03:44.160>of <00:03:44.360>that.

62
00:03:47.860 --> 00:03:51.600
<00:03:47.860>No <00:03:47.987>failure <00:03:48.114>is <00:03:48.241>ever <00:03:48.369>shown <00:03:48.496>in <00:03:48.623>a <00:03:48.750>demo. <00:03:48.878>And <00:03:49.005>the <00:03:49.132>best <00:03:49.260>part

63
00:03:51.640 --> 00:03:54.549
<00:03:51.640>about <00:03:51.873>the <00:03:52.106>demo, <00:03:52.339>"If <00:03:52.573>this <00:03:52.806>is <00:03:53.040>working,

64
00:03:54.800 --> 00:03:58.720
<00:03:54.800>imagine <00:03:55.000>what <00:03:55.200>would <00:03:55.400>happen <00:03:55.600>if <00:03:55.800>we <00:03:56.000>do <00:03:56.200>this."

65
00:03:59.280 --> 00:04:01.020
<00:03:59.280>And <00:03:59.513>that's <00:03:59.746>how <00:03:59.980>generally <00:04:00.213>the <00:04:00.446>demos <00:04:00.680>go.

66
00:04:02.080 --> 00:04:05.870
<00:04:02.080>It <00:04:02.207>proves <00:04:02.334>that <00:04:02.461>a <00:04:02.589>happy <00:04:02.716>path <00:04:02.843>exists, <00:04:02.970>but <00:04:03.098>it <00:04:03.225>says <00:04:03.352>nothing <00:04:03.480>about

67
00:04:05.920 --> 00:04:09.650
<00:04:05.920>whether <00:04:06.059>feature <00:04:06.200>survives <00:04:06.339>contact <00:04:06.480>with <00:04:06.619>users, <00:04:06.760>and <00:04:06.899>which <00:04:07.040>is <00:04:07.179>the <00:04:07.320>most

68
00:04:09.680 --> 00:04:10.440
<00:04:09.680>critical <00:04:10.364>part.

69
00:04:12.260 --> 00:04:14.900
<00:04:12.260>And <00:04:12.435>the <00:04:12.609>problem <00:04:12.785>is <00:04:12.959>almost <00:04:13.135>never <00:04:13.310>just <00:04:13.484>the <00:04:13.660>model.

70
00:04:16.101 --> 00:04:19.840
<00:04:16.101>We <00:04:16.228>actually <00:04:16.355>put <00:04:16.482>a <00:04:16.610>lot <00:04:16.737>of <00:04:16.864>energy <00:04:16.991>into <00:04:17.119>the <00:04:17.246>model <00:04:17.373>quality, <00:04:17.500>prompt

71
00:04:19.860 --> 00:04:22.019
<00:04:19.860>engineering, <00:04:20.326>maybe <00:04:20.793>even <00:04:21.260>fine-tuning.

72
00:04:23.260 --> 00:04:23.580
But

73
00:04:25.700 --> 00:04:29.600
<00:04:25.700>you <00:04:25.875>don't <00:04:26.050>get <00:04:26.224>drop-off <00:04:26.400>because <00:04:26.575>your <00:04:26.750>model <00:04:26.925>is <00:04:27.099>not

74
00:04:30.380 --> 00:04:33.400
<00:04:30.380>Mythos <00:04:30.613>or <00:04:30.846>Fable <00:04:31.080>and <00:04:31.313>you're <00:04:31.546>using <00:04:31.780>Sonnet.

75
00:04:33.930 --> 00:04:37.700
<00:04:33.930>It's <00:04:34.105>everything <00:04:34.280>around <00:04:34.455>that <00:04:34.630>model, <00:04:34.805>the <00:04:34.980>plumbing, <00:04:35.155>the <00:04:35.330>user

76
00:04:37.740 --> 00:04:38.660
experience,

77
00:04:40.160 --> 00:04:44.100
<00:04:40.159>the <00:04:40.439>user <00:04:40.719>reliance <00:04:40.999>that <00:04:41.280>matters. <00:04:41.559>So

78
00:04:44.960 --> 00:04:48.600
<00:04:44.960>whereas <00:04:45.087>we <00:04:45.214>might <00:04:45.341>be <00:04:45.469>thinking, <00:04:45.596>"Oh, <00:04:45.723>we <00:04:45.850>need <00:04:45.978>to <00:04:46.105>score <00:04:46.232>better <00:04:46.359>on

79
00:04:48.680 --> 00:04:50.980
<00:04:48.680>benchmarks. <00:04:48.913>We <00:04:49.146>need <00:04:49.380>to <00:04:49.613>think <00:04:49.846>about <00:04:50.080>latency,

80
00:04:52.760 --> 00:04:53.870
<00:04:52.760>have <00:04:53.759>different

81
00:04:54.780 --> 00:04:58.720
<00:04:54.780>AB <00:04:54.880>tests <00:04:54.979>of <00:04:55.080>the <00:04:55.179>different <00:04:55.280>prompts." <00:04:55.380>And <00:04:55.479>I'm <00:04:55.580>not <00:04:55.679>saying <00:04:55.780>that's <00:04:55.880>wrong, <00:04:55.979>but <00:04:56.080>at <00:04:56.179>the

82
00:04:58.740 --> 00:05:02.150
<00:04:58.740>end <00:04:58.840>of <00:04:58.940>the <00:04:59.040>day, <00:04:59.140>what <00:04:59.240>the <00:04:59.340>user <00:04:59.440>actually <00:04:59.540>cares <00:04:59.640>about <00:04:59.740>is, <00:04:59.840>"Can <00:04:59.940>I <00:05:00.040>trust <00:05:00.140>this

83
00:05:02.160 --> 00:05:05.820
<00:05:02.160>answer? <00:05:02.360>Does <00:05:02.560>this <00:05:02.760>tool <00:05:02.960>fit <00:05:03.160>into <00:05:03.360>how <00:05:03.560>I

84
00:05:05.880 --> 00:05:06.620
<00:05:05.880>already <00:05:06.546>work?

85
00:05:07.740 --> 00:05:08.790
<00:05:07.740>What <00:05:07.976>happens <00:05:08.212>when <00:05:08.448>it's <00:05:08.685>wrong?

86
00:05:09.680 --> 00:05:13.360
<00:05:09.680>And <00:05:09.796>am <00:05:09.913>I <00:05:10.030>still <00:05:10.146>in <00:05:10.263>control?" <00:05:10.380>So <00:05:10.496>these <00:05:10.613>are <00:05:10.730>the <00:05:10.846>questions <00:05:10.963>that <00:05:11.080>the

87
00:05:13.440 --> 00:05:17.340
<00:05:13.440>users <00:05:13.595>actually <00:05:13.751>care <00:05:13.906>about <00:05:14.062>and <00:05:14.217>keep <00:05:14.373>asking <00:05:14.528>themselves <00:05:14.684>when <00:05:14.840>using

88
00:05:17.380 --> 00:05:17.820
<00:05:17.380>the <00:05:17.776>AI.

89
00:05:19.480 --> 00:05:23.180
<00:05:19.480>One <00:05:19.655>of <00:05:19.830>the <00:05:20.005>biggest <00:05:20.180>myths, <00:05:20.355>hypothesis, <00:05:20.530>or <00:05:20.705>assumption <00:05:20.880>we

90
00:05:23.240 --> 00:05:27.180
<00:05:23.240>engineers <00:05:23.440>or <00:05:23.640>developers <00:05:23.840>have <00:05:24.040>is <00:05:24.240>that <00:05:24.440>the <00:05:24.640>adoption

91
00:05:27.220 --> 00:05:31.080
<00:05:27.220>is <00:05:27.327>low <00:05:27.435>because <00:05:27.543>the <00:05:27.650>product <00:05:27.758>is <00:05:27.866>not <00:05:27.973>as <00:05:28.081>good, <00:05:28.189>and <00:05:28.296>we <00:05:28.404>need <00:05:28.512>a <00:05:28.620>bigger

92
00:05:31.140 --> 00:05:34.420
<00:05:31.140>model, <00:05:31.315>better <00:05:31.490>prompts, <00:05:31.664>more <00:05:31.840>fine-tuning <00:05:32.015>to <00:05:32.190>improve <00:05:32.365>the <00:05:32.539>product.

93
00:05:34.830 --> 00:05:37.080
<00:05:34.830>And <00:05:35.063>that's <00:05:35.296>because <00:05:35.530>it's <00:05:35.763>in <00:05:35.996>our <00:05:36.229>control.

94
00:05:37.860 --> 00:05:41.400
<00:05:37.860>We <00:05:38.015>would <00:05:38.171>rather <00:05:38.326>build <00:05:38.482>new <00:05:38.637>features <00:05:38.793>rather <00:05:38.948>than <00:05:39.104>thinking <00:05:39.260>about

95
00:05:42.660 --> 00:05:45.700
<00:05:42.659>how <00:05:43.010>to <00:05:43.359>grow <00:05:43.710>adoption <00:05:44.059>and

96
00:05:46.000 --> 00:05:47.260
<00:05:46.000>understanding <00:05:46.567>the <00:05:47.134>user.

97
00:05:48.980 --> 00:05:50.620
<00:05:48.980>The <00:05:49.446>reality <00:05:49.913>is <00:05:50.380>actually,

98
00:05:51.780 --> 00:05:54.340
<00:05:51.780>the <00:05:51.979>user <00:05:52.179>generally <00:05:52.380>churns <00:05:52.580>on <00:05:52.780>trust <00:05:52.979>and <00:05:53.179>workflow,

99
00:05:55.540 --> 00:05:57.660
<00:05:55.540>not <00:05:55.740>really <00:05:55.940>on <00:05:56.140>how <00:05:56.340>big <00:05:56.540>the <00:05:56.740>model <00:05:56.940>is.

100
00:05:59.080 --> 00:06:02.876
<00:05:59.080>And <00:05:59.235>to <00:05:59.391>some <00:05:59.546>extent, <00:05:59.702>a <00:05:59.857>better <00:06:00.013>model <00:06:00.168>might <00:06:00.324>Give <00:06:00.479>a

101
00:06:02.916 --> 00:06:06.516
<00:06:02.916>more <00:06:03.091>confident <00:06:03.266>wrong <00:06:03.441>answer, <00:06:03.616>which <00:06:03.791>creates <00:06:03.966>even <00:06:04.141>more <00:06:04.316>problem.

102
00:06:07.136 --> 00:06:10.706
<00:06:07.136>And <00:06:07.336>the <00:06:07.536>failure <00:06:07.736>generally <00:06:07.936>resides <00:06:08.136>at <00:06:08.336>the <00:06:08.536>seams

103
00:06:11.296 --> 00:06:15.156
<00:06:11.296>of <00:06:11.412>the <00:06:11.529>handoffs <00:06:11.646>are <00:06:11.762>doing, <00:06:11.879>where's <00:06:11.996>the <00:06:12.112>control <00:06:12.229>lying, <00:06:12.346>is <00:06:12.462>the <00:06:12.579>feedback <00:06:12.695>being

104
00:06:15.256 --> 00:06:15.956
<00:06:15.256>taken <00:06:15.886>in,

105
00:06:17.296 --> 00:06:20.156
<00:06:17.296>which <00:06:17.496>are <00:06:17.695>always <00:06:17.896>model <00:06:18.096>agnostic, <00:06:18.296>which <00:06:18.496>are <00:06:18.695>more

106
00:06:21.096 --> 00:06:24.496
<00:06:21.096>part <00:06:21.271>of <00:06:21.446>your <00:06:21.621>product <00:06:21.796>rather <00:06:21.971>than <00:06:22.146>part <00:06:22.321>of <00:06:22.496>the

107
00:06:25.496 --> 00:06:26.816
<00:06:25.496>model <00:06:25.892>that <00:06:26.287>you're <00:06:26.683>using.

108
00:06:28.156 --> 00:06:32.126
<00:06:28.156>And <00:06:28.436>generally, <00:06:28.716>a <00:06:28.996>better <00:06:29.276>model <00:06:29.556>raises

109
00:06:32.156 --> 00:06:34.486
<00:06:32.156>the <00:06:32.436>ceiling, <00:06:32.716>but <00:06:32.996>doesn't <00:06:33.276>do <00:06:33.556>anything

110
00:06:36.356 --> 00:06:40.336
<00:06:36.356>to <00:06:36.496>improve <00:06:36.635>the <00:06:36.776>floor, <00:06:36.916>and <00:06:37.056>the <00:06:37.195>adoption <00:06:37.336>generally <00:06:37.476>dies <00:06:37.616>on <00:06:37.756>the

111
00:06:40.356 --> 00:06:40.656
floor.

112
00:06:42.916 --> 00:06:46.576
<00:06:42.916>So <00:06:43.023>now <00:06:43.131>we're <00:06:43.239>going <00:06:43.346>to <00:06:43.454>talk <00:06:43.562>about <00:06:43.669>the <00:06:43.777>five <00:06:43.885>failure <00:06:43.992>modes <00:06:44.100>that <00:06:44.208>I've <00:06:44.316>seen

113
00:06:48.676 --> 00:06:52.176
<00:06:48.676>in <00:06:48.876>production, <00:06:49.075>where <00:06:49.276>things <00:06:49.476>actually <00:06:49.676>break <00:06:49.876>and <00:06:50.075>things

114
00:06:52.186 --> 00:06:55.506
<00:06:52.186>actually <00:06:52.536>create <00:06:52.885>issues <00:06:53.236>in <00:06:53.585>adoption.

115
00:06:57.636 --> 00:06:59.256
<00:06:57.636>Number <00:06:58.102>one, <00:06:58.569>workflow <00:06:59.036>fit.

116
00:07:00.536 --> 00:07:00.736
Does

117
00:07:01.656 --> 00:07:05.276
<00:07:01.656>your <00:07:01.811>new <00:07:01.967>AI <00:07:02.122>feature <00:07:02.278>add <00:07:02.433>steps <00:07:02.589>instead <00:07:02.744>of <00:07:02.900>removing <00:07:03.056>them?

118
00:07:05.976 --> 00:07:09.756
<00:07:05.976>Number <00:07:06.116>two, <00:07:06.256>trust. <00:07:06.396>Can <00:07:06.536>user <00:07:06.676>tell <00:07:06.816>when <00:07:06.956>to <00:07:07.096>believe <00:07:07.236>it <00:07:07.376>or

119
00:07:09.836 --> 00:07:11.636
<00:07:09.836>not? <00:07:10.186>Number <00:07:10.536>three, <00:07:10.886>handoff <00:07:11.236>design.

120
00:07:11.956 --> 00:07:15.796
<00:07:11.956>Is <00:07:12.111>there <00:07:12.267>a <00:07:12.422>graceful <00:07:12.578>path <00:07:12.733>when <00:07:12.889>the <00:07:13.044>AI <00:07:13.200>gives <00:07:13.356>up?

121
00:07:15.836 --> 00:07:17.146
<00:07:15.836>Number <00:07:16.229>four, <00:07:16.622>user <00:07:17.015>control.

122
00:07:18.236 --> 00:07:22.176
<00:07:18.236>Does <00:07:18.376>it <00:07:18.515>act <00:07:18.656>when <00:07:18.796>it <00:07:18.936>should <00:07:19.075>only <00:07:19.216>suggest <00:07:19.356>and <00:07:19.496>vice <00:07:19.635>versa?

123
00:07:22.876 --> 00:07:24.266
<00:07:22.876>Number <00:07:23.293>five, <00:07:23.710>product <00:07:24.127>discipline.

124
00:07:24.416 --> 00:07:27.976
<00:07:24.416>Is <00:07:24.532>there <00:07:24.649>feedback <00:07:24.766>loops <00:07:24.882>in <00:07:24.999>there <00:07:25.116>to <00:07:25.232>improve <00:07:25.349>your <00:07:25.466>product <00:07:25.582>in <00:07:25.699>the <00:07:25.816>future?

125
00:07:29.196 --> 00:07:32.856
<00:07:29.196>So <00:07:29.351>coming <00:07:29.507>into <00:07:29.662>failure <00:07:29.818>mode <00:07:29.973>number <00:07:30.129>one, <00:07:30.284>which <00:07:30.440>is <00:07:30.596>workflow

126
00:07:32.976 --> 00:07:33.376
fit.

127
00:07:35.156 --> 00:07:38.996
<00:07:35.156>This <00:07:35.256>is <00:07:35.356>where <00:07:35.456>we <00:07:35.556>need <00:07:35.656>to <00:07:35.756>think <00:07:35.856>whether <00:07:35.956>we <00:07:36.056>are <00:07:36.156>adding <00:07:36.256>more <00:07:36.356>work <00:07:36.456>instead <00:07:36.556>of

128
00:07:39.056 --> 00:07:40.696
<00:07:39.056>reducing <00:07:39.335>the <00:07:39.616>work <00:07:39.895>for <00:07:40.176>the <00:07:40.455>users.

129
00:07:41.656 --> 00:07:42.626
<00:07:41.656>For <00:07:42.529>example,

130
00:07:44.076 --> 00:07:46.246
<00:07:44.076>let's <00:07:44.309>take <00:07:44.542>an <00:07:44.776>example <00:07:45.009>of <00:07:45.242>writing <00:07:45.476>emails.

131
00:07:48.936 --> 00:07:52.196
<00:07:48.936>Right <00:07:49.169>now <00:07:49.402>we <00:07:49.635>have <00:07:49.869>two <00:07:50.102>options. <00:07:50.335>Number

132
00:07:52.316 --> 00:07:54.636
<00:07:52.316>one, <00:07:52.782>we <00:07:53.249>can <00:07:53.716>use

133
00:07:55.336 --> 00:07:58.996
<00:07:55.336>ChatGPT, <00:07:55.511>wherein <00:07:55.686>we <00:07:55.861>have <00:07:56.036>to <00:07:56.211>go <00:07:56.386>to <00:07:56.561>our <00:07:56.736>email,

134
00:07:59.376 --> 00:08:00.376
<00:07:59.376>copy-paste <00:07:59.825>the <00:08:00.275>data,

135
00:08:01.216 --> 00:08:03.596
<00:08:01.216>take <00:08:01.496>it <00:08:01.776>to <00:08:02.056>ChatGPT, <00:08:02.336>paste <00:08:02.616>it,

136
00:08:04.696 --> 00:08:07.976
<00:08:04.696>get <00:08:04.812>it <00:08:04.929>to <00:08:05.046>write <00:08:05.162>an <00:08:05.279>answer, <00:08:05.396>give <00:08:05.512>it <00:08:05.629>context, <00:08:05.746>and <00:08:05.862>bring <00:08:05.979>it <00:08:06.096>back.

137
00:08:08.336 --> 00:08:09.896
<00:08:08.336>So <00:08:08.686>that's <00:08:09.036>option <00:08:09.386>number <00:08:09.736>one.

138
00:08:11.355 --> 00:08:14.556
<00:08:11.355>At <00:08:11.462>some <00:08:11.570>point <00:08:11.678>of <00:08:11.785>time, <00:08:11.893>for <00:08:12.001>smaller <00:08:12.108>tasks, <00:08:12.216>you're <00:08:12.324>just <00:08:12.431>going <00:08:12.539>to <00:08:12.647>be <00:08:12.755>like,

139
00:08:15.316 --> 00:08:18.816
<00:08:15.316>"I'll <00:08:15.403>write <00:08:15.491>it <00:08:15.578>myself." <00:08:15.666>It <00:08:15.753>doesn't <00:08:15.840>need <00:08:15.928>that <00:08:16.015>much <00:08:16.103>work <00:08:16.191>because <00:08:16.278>I <00:08:16.366>need <00:08:16.453>to <00:08:16.541>go <00:08:16.628>to <00:08:16.715>a

140
00:08:18.856 --> 00:08:20.246
<00:08:18.856>separate <00:08:19.168>tab, <00:08:19.481>bring <00:08:19.794>it <00:08:20.107>back,

141
00:08:21.376 --> 00:08:23.936
<00:08:21.376>explain <00:08:21.726>the <00:08:22.075>entire <00:08:22.426>context. <00:08:22.775>But

142
00:08:25.556 --> 00:08:27.996
<00:08:25.556>what <00:08:25.835>do <00:08:26.116>we <00:08:26.395>get <00:08:26.676>with <00:08:26.955>Gemini?

143
00:08:28.116 --> 00:08:31.696
<00:08:28.116>We <00:08:28.466>write <00:08:28.816>emails <00:08:29.166>on <00:08:29.515>Gmail

144
00:08:32.036 --> 00:08:33.817
<00:08:32.036>or <00:08:32.386>whichever <00:08:32.736>provider <00:08:33.086>you <00:08:33.436>have.

145
00:08:34.636 --> 00:08:38.516
<00:08:34.635>Gemini <00:08:34.775>is <00:08:34.915>now <00:08:35.055>integrated <00:08:35.195>in <00:08:35.336>Gmail, <00:08:35.476>so <00:08:35.616>it <00:08:35.756>can <00:08:35.895>read <00:08:36.035>basically

146
00:08:38.557 --> 00:08:41.645
<00:08:38.557>the <00:08:38.757>entire <00:08:38.957>chat <00:08:39.157>or <00:08:39.357>the <00:08:39.557>entire <00:08:39.757>email <00:08:39.957>history

147
00:08:42.696 --> 00:08:46.416
<00:08:42.696>of <00:08:42.836>that <00:08:42.976>particular <00:08:43.116>thread, <00:08:43.256>in <00:08:43.396>that <00:08:43.536>particular <00:08:43.676>thread, <00:08:43.816>and <00:08:43.956>get <00:08:44.096>the

148
00:08:46.456 --> 00:08:47.826
<00:08:46.456>context <00:08:47.689>and,

149
00:08:49.516 --> 00:08:53.116
<00:08:49.516>in <00:08:49.632>three <00:08:49.749>words <00:08:49.866>or <00:08:49.982>more, <00:08:50.099>if <00:08:50.216>you <00:08:50.332>type <00:08:50.449>it <00:08:50.565>what <00:08:50.682>you <00:08:50.799>want <00:08:50.915>to

150
00:08:53.126 --> 00:08:55.576
<00:08:53.126>answer, <00:08:53.406>it <00:08:53.685>creates <00:08:53.966>an <00:08:54.246>entire <00:08:54.526>email.

151
00:08:56.196 --> 00:08:59.296
<00:08:56.196>Now, <00:08:56.396>I <00:08:56.596>did <00:08:56.796>not <00:08:56.996>have <00:08:57.196>to <00:08:57.396>switch <00:08:57.596>tabs.

152
00:09:00.436 --> 00:09:02.336
<00:09:00.436>I <00:09:00.636>did <00:09:00.836>not <00:09:01.036>have <00:09:01.236>to <00:09:01.436>explain <00:09:01.636>another <00:09:01.836>model

153
00:09:03.376 --> 00:09:05.816
<00:09:03.376>what <00:09:03.576>I <00:09:03.776>meant <00:09:03.976>or <00:09:04.175>what <00:09:04.376>is <00:09:04.576>the <00:09:04.776>context.

154
00:09:06.216 --> 00:09:08.216
<00:09:06.216>Over <00:09:06.566>here, <00:09:06.916>everything <00:09:07.266>is <00:09:07.616>done.

155
00:09:09.356 --> 00:09:13.256
<00:09:09.356>The <00:09:09.511>AI <00:09:09.667>feature <00:09:09.822>is <00:09:09.978>in <00:09:10.133>where <00:09:10.289>my <00:09:10.444>existing <00:09:10.600>workflow <00:09:10.756>lies.

156
00:09:13.316 --> 00:09:16.316
<00:09:13.316>So <00:09:13.471>they <00:09:13.627>have <00:09:13.782>actually <00:09:13.938>reduced <00:09:14.093>my <00:09:14.249>number <00:09:14.404>of <00:09:14.560>steps <00:09:14.716>rather

157
00:09:16.356 --> 00:09:20.076
<00:09:16.356>than <00:09:16.589>increasing <00:09:16.822>my <00:09:17.056>numbers <00:09:17.289>of <00:09:17.522>steps <00:09:17.756>or

158
00:09:20.096 --> 00:09:22.196
<00:09:20.096>taking <00:09:20.446>me <00:09:20.796>to <00:09:21.146>another <00:09:21.496>destination.

159
00:09:22.816 --> 00:09:26.356
<00:09:22.816>So <00:09:22.971>that's <00:09:23.127>the <00:09:23.282>difference <00:09:23.438>between <00:09:23.593>having <00:09:23.749>a <00:09:23.904>bolt-on <00:09:24.060>or <00:09:24.216>a

160
00:09:26.456 --> 00:09:29.736
<00:09:26.456>built-in <00:09:26.611>product. <00:09:26.767>So <00:09:26.922>a <00:09:27.078>bolt-on <00:09:27.233>would <00:09:27.389>be <00:09:27.544>something <00:09:27.700>like <00:09:27.856>using

161
00:09:29.756 --> 00:09:30.556
ChatGPT

162
00:09:32.156 --> 00:09:35.596
<00:09:32.156>for <00:09:32.295>writing <00:09:32.435>emails, <00:09:32.575>whereas <00:09:32.715>a <00:09:32.856>built-in <00:09:32.996>is <00:09:33.136>having <00:09:33.276>Gemini <00:09:33.415>in <00:09:33.555>Gmail.

163
00:09:36.996 --> 00:09:40.596
<00:09:36.996>So <00:09:37.112>what <00:09:37.229>we <00:09:37.346>want <00:09:37.462>to <00:09:37.579>do <00:09:37.696>as <00:09:37.812>a <00:09:37.929>design <00:09:38.045>principle <00:09:38.162>is <00:09:38.279>go <00:09:38.396>for

164
00:09:40.636 --> 00:09:44.596
<00:09:40.636>built-ins. <00:09:40.836>So <00:09:41.036>if <00:09:41.236>the <00:09:41.435>data <00:09:41.636>lies <00:09:41.836>in <00:09:42.036>Salesforce,

165
00:09:44.636 --> 00:09:48.536
<00:09:44.636>we <00:09:44.791>need <00:09:44.947>to <00:09:45.102>have <00:09:45.258>an <00:09:45.413>AI <00:09:45.569>feature <00:09:45.724>that <00:09:45.880>works <00:09:46.036>on

166
00:09:48.576 --> 00:09:50.816
<00:09:48.576>Salesforce <00:09:49.276>rather <00:09:49.976>than

167
00:09:52.416 --> 00:09:55.496
<00:09:52.415>connecting, <00:09:52.766>creating <00:09:53.116>another <00:09:53.465>dashboard, <00:09:53.815>giving

168
00:09:56.556 --> 00:10:00.436
<00:09:56.556>user <00:09:56.696>education <00:09:56.836>and <00:09:56.976>training <00:09:57.116>another <00:09:57.256>dashboard, <00:09:57.396>which <00:09:57.536>all <00:09:57.676>kind <00:09:57.816>of <00:09:57.956>creates

169
00:10:00.456 --> 00:10:03.656
<00:10:00.456>extra <00:10:00.631>work, <00:10:00.806>and <00:10:00.981>people <00:10:01.156>do <00:10:01.331>not <00:10:01.506>like <00:10:01.681>extra <00:10:01.856>work.

170
00:10:04.796 --> 00:10:07.175
<00:10:04.796>Coming <00:10:05.029>to <00:10:05.262>failure <00:10:05.496>mode <00:10:05.729>number <00:10:05.962>two, <00:10:06.196>trust.

171
00:10:08.256 --> 00:10:09.956
<00:10:08.256>Now, <00:10:08.606>this <00:10:08.956>is <00:10:09.305>very <00:10:09.656>interesting.

172
00:10:12.136 --> 00:10:15.776
<00:10:12.136>With <00:10:12.486>AI, <00:10:12.836>people <00:10:13.185>are <00:10:13.536>already

173
00:10:15.956 --> 00:10:18.916
<00:10:15.956>careful <00:10:16.156>about <00:10:16.356>hallucinations. <00:10:16.556>They <00:10:16.756>know <00:10:16.956>about <00:10:17.156>the <00:10:17.356>risks.

174
00:10:19.996 --> 00:10:23.396
<00:10:19.996>So <00:10:20.136>trust <00:10:20.276>in <00:10:20.415>AI <00:10:20.555>takes <00:10:20.696>a <00:10:20.836>lot <00:10:20.976>of <00:10:21.116>time <00:10:21.256>to <00:10:21.396>build,

175
00:10:24.296 --> 00:10:28.236
<00:10:24.296>whereas-- <00:10:24.396>and <00:10:24.496>at <00:10:24.596>the <00:10:24.696>same <00:10:24.796>time, <00:10:24.896>we <00:10:24.996>need <00:10:25.096>to <00:10:25.196>be <00:10:25.296>aware <00:10:25.396>that <00:10:25.496>trust <00:10:25.596>is <00:10:25.696>asymmetric.

176
00:10:28.696 --> 00:10:32.316
<00:10:28.696>So <00:10:28.871>it <00:10:29.046>takes <00:10:29.221>a <00:10:29.396>lot <00:10:29.571>of <00:10:29.746>time <00:10:29.921>to <00:10:30.096>build,

177
00:10:33.336 --> 00:10:36.596
<00:10:33.336>and <00:10:33.511>one <00:10:33.686>or <00:10:33.861>two <00:10:34.036>mistakes <00:10:34.211>could <00:10:34.386>drop <00:10:34.561>it <00:10:34.736>immediately.

178
00:10:37.676 --> 00:10:38.496
<00:10:37.676>And <00:10:38.045>to <00:10:38.414>some

179
00:10:39.376 --> 00:10:41.196
<00:10:39.376>level, <00:10:39.842>what <00:10:40.309>is <00:10:40.776>trust?

180
00:10:42.096 --> 00:10:44.476
<00:10:42.096>Trust <00:10:42.446>is <00:10:42.796>actually <00:10:43.146>calibrated <00:10:43.496>reliance.

181
00:10:45.356 --> 00:10:45.696
So

182
00:10:48.876 --> 00:10:52.016
<00:10:48.876>you <00:10:49.076>need <00:10:49.276>to <00:10:49.476>be <00:10:49.675>able <00:10:49.876>to <00:10:50.076>build <00:10:50.276>that

183
00:10:52.516 --> 00:10:56.016
<00:10:52.516>reliance <00:10:52.632>of <00:10:52.749>the <00:10:52.866>user <00:10:52.982>on <00:10:53.099>your <00:10:53.216>product. <00:10:53.332>And <00:10:53.449>how <00:10:53.565>would <00:10:53.682>you <00:10:53.799>do <00:10:53.915>that?

184
00:10:56.096 --> 00:10:59.276
<00:10:56.096>There <00:10:56.296>are <00:10:56.496>some <00:10:56.696>of <00:10:56.896>the <00:10:57.096>obvious <00:10:57.296>ways <00:10:57.496>wherein

185
00:11:00.436 --> 00:11:00.956
<00:11:00.436>you <00:11:00.904>show

186
00:11:02.336 --> 00:11:06.136
<00:11:02.336>basically <00:11:02.511>where <00:11:02.686>you're <00:11:02.861>taking <00:11:03.036>the <00:11:03.211>information <00:11:03.386>from, <00:11:03.561>giving <00:11:03.736>the

187
00:11:06.196 --> 00:11:10.096
<00:11:06.196>user <00:11:06.351>links, <00:11:06.507>traces, <00:11:06.662>so <00:11:06.818>that <00:11:06.973>the <00:11:07.129>user <00:11:07.284>is <00:11:07.440>able <00:11:07.596>to

188
00:11:10.156 --> 00:11:11.716
<00:11:10.156>verify <00:11:10.622>it <00:11:11.089>for <00:11:11.555>themselves.

189
00:11:12.916 --> 00:11:13.236
And

190
00:11:14.176 --> 00:11:16.056
<00:11:14.176>you <00:11:14.526>make <00:11:14.876>mistakes <00:11:15.226>cheap. <00:11:15.576>So,

191
00:11:17.116 --> 00:11:20.756
<00:11:17.116>for <00:11:17.243>example, <00:11:17.370>if <00:11:17.497>I <00:11:17.625>do <00:11:17.752>not <00:11:17.879>like <00:11:18.006>the <00:11:18.134>code <00:11:18.261>Claude <00:11:18.388>Code <00:11:18.516>is

192
00:11:20.816 --> 00:11:24.596
<00:11:20.816>writing, <00:11:20.943>I'm <00:11:21.070>able <00:11:21.197>to <00:11:21.325>go <00:11:21.452>back <00:11:21.579>to <00:11:21.706>the <00:11:21.834>previous <00:11:21.961>commit <00:11:22.088>and <00:11:22.216>revert

193
00:11:24.756 --> 00:11:27.056
<00:11:24.756>all <00:11:24.989>the <00:11:25.222>changes. <00:11:25.456>I'm <00:11:25.689>able <00:11:25.922>to <00:11:26.156>stop

194
00:11:28.436 --> 00:11:32.356
<00:11:28.436>it <00:11:28.536>mid-work <00:11:28.636>and <00:11:28.736>tell <00:11:28.836>it, <00:11:28.936>"No, <00:11:29.036>you're <00:11:29.136>wrong," <00:11:29.236>and <00:11:29.336>have <00:11:29.436>it <00:11:29.536>revert <00:11:29.636>all <00:11:29.736>the <00:11:29.836>changes.

195
00:11:32.936 --> 00:11:35.016
<00:11:32.936>So <00:11:33.169>even <00:11:33.402>if <00:11:33.636>it <00:11:33.869>makes <00:11:34.102>a <00:11:34.336>mistake,

196
00:11:36.096 --> 00:11:39.976
<00:11:36.096>it's <00:11:36.223>cheap. <00:11:36.350>And <00:11:36.477>that's <00:11:36.605>why <00:11:36.732>we <00:11:36.859>should <00:11:36.986>not <00:11:37.114>give <00:11:37.241>it <00:11:37.368>our <00:11:37.496>AWS

197
00:11:40.776 --> 00:11:44.336
<00:11:40.776>keys <00:11:40.915>so <00:11:41.055>it <00:11:41.195>cannot <00:11:41.335>deploy, <00:11:41.476>because <00:11:41.616>once <00:11:41.756>it's <00:11:41.896>deployed <00:11:42.036>a <00:11:42.175>change,

198
00:11:46.096 --> 00:11:48.016
<00:11:46.096>then <00:11:46.562>the <00:11:47.029>costs <00:11:47.496>become

199
00:11:48.136 --> 00:11:51.936
<00:11:48.136>higher. <00:11:48.486>Failure <00:11:48.836>mode <00:11:49.185>number <00:11:49.536>three,

200
00:11:52.156 --> 00:11:52.996
<00:11:52.156>handoff <00:11:52.912>design.

201
00:11:53.956 --> 00:11:57.456
<00:11:53.956>So <00:11:54.131>that <00:11:54.306>was <00:11:54.481>only <00:11:54.656>show <00:11:54.831>successes, <00:11:55.006>whereas <00:11:55.181>in <00:11:55.356>production,

202
00:11:57.696 --> 00:12:01.276
<00:11:57.696>it's <00:11:57.836>full <00:11:57.976>of <00:11:58.116>low <00:11:58.256>confidence, <00:11:58.396>ambiguous, <00:11:58.536>and <00:11:58.676>a <00:11:58.816>lot <00:11:58.956>of <00:11:59.096>hope

203
00:12:01.356 --> 00:12:01.896
cases.

204
00:12:02.956 --> 00:12:06.396
<00:12:02.956>So <00:12:03.096>we <00:12:03.236>need <00:12:03.376>to <00:12:03.516>know <00:12:03.656>where <00:12:03.796>we <00:12:03.936>need <00:12:04.076>humans <00:12:04.216>as <00:12:04.356>a

205
00:12:06.436 --> 00:12:08.196
<00:12:06.436>fallback <00:12:06.716>and <00:12:06.996>where <00:12:07.276>AI <00:12:07.556>can <00:12:07.836>act.

206
00:12:09.368 --> 00:12:12.828
<00:12:09.368>So <00:12:09.495>we <00:12:09.622>can <00:12:09.749>broadly <00:12:09.877>categorize <00:12:10.004>based <00:12:10.131>on <00:12:10.258>the <00:12:10.386>confidence <00:12:10.513>level <00:12:10.640>of <00:12:10.768>the

207
00:12:12.888 --> 00:12:13.488
request

208
00:12:15.108 --> 00:12:18.768
<00:12:15.108>into <00:12:15.235>four <00:12:15.362>different <00:12:15.489>categories. <00:12:15.617>Number <00:12:15.744>one, <00:12:15.871>we <00:12:15.998>are <00:12:16.126>confident <00:12:16.253>about <00:12:16.380>this <00:12:16.507>request.

209
00:12:18.848 --> 00:12:22.068
<00:12:18.848>Number <00:12:19.048>two, <00:12:19.247>we <00:12:19.448>are <00:12:19.647>uncertain. <00:12:19.848>Number <00:12:20.048>three, <00:12:20.247>ambiguous.

210
00:12:22.208 --> 00:12:25.328
<00:12:22.208>And <00:12:22.315>number <00:12:22.423>four, <00:12:22.531>out <00:12:22.638>of <00:12:22.746>scope, <00:12:22.854>wherein <00:12:22.961>we <00:12:23.069>should <00:12:23.177>definitely <00:12:23.284>route <00:12:23.392>to <00:12:23.500>a <00:12:23.608>human.

211
00:12:27.248 --> 00:12:30.188
<00:12:27.248>And <00:12:27.481>we <00:12:27.714>can <00:12:27.948>easily <00:12:28.181>do <00:12:28.414>it <00:12:28.648>by

212
00:12:31.788 --> 00:12:33.028
<00:12:31.788>having <00:12:32.346>confidence <00:12:32.904>scores.

213
00:12:34.728 --> 00:12:38.218
<00:12:34.728>So <00:12:34.928>for <00:12:35.127>example, <00:12:35.328>for <00:12:35.527>number <00:12:35.728>one, <00:12:35.928>wherein <00:12:36.127>I've

214
00:12:38.248 --> 00:12:40.008
<00:12:38.248>been <00:12:38.598>coding <00:12:38.948>with <00:12:39.298>Claude <00:12:39.648>Code,

215
00:12:40.888 --> 00:12:44.668
<00:12:40.888>I <00:12:41.015>wanted <00:12:41.142>to <00:12:41.269>add <00:12:41.397>an <00:12:41.524>extra <00:12:41.651>button, <00:12:41.778>which <00:12:41.906>should <00:12:42.033>CTA, <00:12:42.160>which <00:12:42.288>should

216
00:12:44.688 --> 00:12:47.908
<00:12:44.688>redirect <00:12:45.038>to <00:12:45.388>this <00:12:45.738>page. <00:12:46.088>It's

217
00:12:49.028 --> 00:12:52.228
<00:12:49.028>very <00:12:49.144>clear, <00:12:49.261>very <00:12:49.378>easy <00:12:49.494>request, <00:12:49.611>so <00:12:49.728>it <00:12:49.844>should <00:12:49.961>be <00:12:50.078>able <00:12:50.194>to <00:12:50.311>do <00:12:50.428>it.

218
00:12:52.388 --> 00:12:56.348
<00:12:52.388>And <00:12:52.515>it's <00:12:52.642>a <00:12:52.769>small <00:12:52.897>feature, <00:12:53.024>so <00:12:53.151>Claude <00:12:53.278>Code <00:12:53.406>can <00:12:53.533>decide, <00:12:53.660>yes, <00:12:53.788>it

219
00:12:56.428 --> 00:12:59.948
<00:12:56.428>knows <00:12:56.555>it, <00:12:56.682>and <00:12:56.809>it <00:12:56.937>can <00:12:57.064>act <00:12:57.191>on <00:12:57.318>my <00:12:57.446>behalf, <00:12:57.573>and <00:12:57.700>it <00:12:57.828>can

220
00:12:59.988 --> 00:13:02.488
<00:12:59.988>create <00:13:00.268>that <00:13:00.548>feature <00:13:00.828>and <00:13:01.108>commit <00:13:01.388>it.

221
00:13:03.588 --> 00:13:07.568
<00:13:03.588>Whereas <00:13:03.938>something <00:13:04.288>like, <00:13:04.637>"Can <00:13:04.988>you

222
00:13:07.608 --> 00:13:11.368
<00:13:07.608>pay <00:13:07.808>this <00:13:08.007>person <00:13:08.208>$200 <00:13:08.407>or <00:13:08.608>refund <00:13:08.808>this <00:13:09.007>person

223
00:13:11.408 --> 00:13:12.168
$200?"

224
00:13:13.168 --> 00:13:14.908
<00:13:13.168>Those <00:13:13.368>should <00:13:13.568>be <00:13:13.768>completely <00:13:13.968>out <00:13:14.168>of <00:13:14.368>the <00:13:14.568>scope.

225
00:13:16.648 --> 00:13:20.408
<00:13:16.648>And <00:13:16.788>because <00:13:16.928>it's <00:13:17.068>a <00:13:17.208>payment <00:13:17.348>one, <00:13:17.488>it's <00:13:17.628>a <00:13:17.768>sensitive <00:13:17.908>one, <00:13:18.048>we

226
00:13:20.448 --> 00:13:24.368
<00:13:20.448>don't <00:13:20.648>want <00:13:20.848>to <00:13:21.048>expose <00:13:21.247>the <00:13:21.448>PII. <00:13:21.648>We <00:13:21.848>do

227
00:13:24.428 --> 00:13:25.468
<00:13:24.428>not <00:13:24.896>want <00:13:25.363>to

228
00:13:28.508 --> 00:13:32.068
<00:13:28.508>expose <00:13:28.708>any <00:13:28.908>of <00:13:29.108>the <00:13:29.308>card <00:13:29.508>details <00:13:29.708>or <00:13:29.908>payment

229
00:13:32.148 --> 00:13:35.708
<00:13:32.148>details <00:13:32.498>to <00:13:32.848>a <00:13:33.198>third-party <00:13:33.548>server,

230
00:13:35.788 --> 00:13:39.248
<00:13:35.788>like <00:13:35.988>with <00:13:36.188>OpenAI, <00:13:36.388>Anthropic, <00:13:36.588>or <00:13:36.788>any <00:13:36.988>of <00:13:37.188>the

231
00:13:39.528 --> 00:13:42.188
<00:13:39.528>AIs <00:13:39.655>that <00:13:39.782>you're <00:13:39.909>using. <00:13:40.037>So <00:13:40.164>that <00:13:40.291>should <00:13:40.418>be <00:13:40.546>completely <00:13:40.673>out <00:13:40.800>of <00:13:40.928>scope.

232
00:13:42.648 --> 00:13:46.588
<00:13:42.648>So <00:13:42.764>over <00:13:42.881>there, <00:13:42.998>a <00:13:43.114>human <00:13:43.231>should <00:13:43.348>take <00:13:43.464>over, <00:13:43.581>and <00:13:43.698>the <00:13:43.814>model <00:13:43.931>should <00:13:44.048>confidently

233
00:13:47.588 --> 00:13:50.308
<00:13:47.588>say, <00:13:47.728>"This <00:13:47.867>is <00:13:48.007>out <00:13:48.147>of <00:13:48.288>the <00:13:48.428>scope," <00:13:48.568>and <00:13:48.708>reject <00:13:48.848>the <00:13:48.988>request.

234
00:13:51.108 --> 00:13:52.818
<00:13:51.108>And <00:13:51.808>based <00:13:52.507>on

235
00:13:54.648 --> 00:13:58.488
<00:13:54.648>the <00:13:54.741>level <00:13:54.834>of <00:13:54.928>request <00:13:55.021>or <00:13:55.114>based <00:13:55.208>on <00:13:55.301>the <00:13:55.394>confidence <00:13:55.488>of <00:13:55.581>the <00:13:55.674>model <00:13:55.768>to <00:13:55.861>be <00:13:55.954>able <00:13:56.048>to

236
00:13:58.608 --> 00:13:59.928
<00:13:58.608>do <00:13:59.202>a <00:13:59.796>particular

237
00:14:00.888 --> 00:14:01.058
action,

238
00:14:01.928 --> 00:14:05.508
<00:14:01.928>we <00:14:02.161>should <00:14:02.394>choose <00:14:02.628>one <00:14:02.861>of <00:14:03.094>these <00:14:03.328>categories

239
00:14:07.108 --> 00:14:08.768
<00:14:07.108>wherein <00:14:07.458>the <00:14:07.808>model <00:14:08.157>should <00:14:08.507>respond.

240
00:14:10.708 --> 00:14:13.708
<00:14:10.708>So <00:14:10.908>failure <00:14:11.108>mode <00:14:11.308>number <00:14:11.507>four <00:14:11.708>is <00:14:11.908>user <00:14:12.108>control.

241
00:14:15.208 --> 00:14:15.508
And

242
00:14:16.468 --> 00:14:20.408
<00:14:16.468>I <00:14:16.595>believe <00:14:16.722>that <00:14:16.849>every <00:14:16.977>AI <00:14:17.104>action <00:14:17.231>sits <00:14:17.358>somewhere <00:14:17.486>on <00:14:17.613>the <00:14:17.740>spectrum <00:14:17.867>from

243
00:14:20.488 --> 00:14:21.628
<00:14:20.488>suggesting <00:14:21.001>to <00:14:21.514>acting.

244
00:14:25.808 --> 00:14:27.488
<00:14:25.808>And <00:14:26.158>it <00:14:26.508>all <00:14:26.858>depends <00:14:27.208>on

245
00:14:29.028 --> 00:14:30.148
<00:14:29.028>what's <00:14:29.364>a <00:14:29.700>low <00:14:30.036>risk

246
00:14:30.988 --> 00:14:34.868
<00:14:30.988>to <00:14:31.143>a <00:14:31.299>high-risk <00:14:31.454>action. <00:14:31.610>So <00:14:31.765>low <00:14:31.921>risk <00:14:32.076>are <00:14:32.232>like, <00:14:32.388>for

247
00:14:34.928 --> 00:14:37.648
<00:14:34.928>example, <00:14:35.278>ChatGPT <00:14:35.628>informing <00:14:35.978>you <00:14:36.328>about

248
00:14:38.968 --> 00:14:40.768
<00:14:38.968>any <00:14:39.201>of <00:14:39.434>the <00:14:39.668>questions <00:14:39.901>that <00:14:40.134>you <00:14:40.367>have

249
00:14:41.628 --> 00:14:44.708
<00:14:41.628>or <00:14:41.768>suggesting <00:14:41.908>you <00:14:42.048>that <00:14:42.188>you <00:14:42.328>should <00:14:42.468>probably <00:14:42.608>go <00:14:42.748>to <00:14:42.888>this <00:14:43.028>restaurant.

250
00:14:45.288 --> 00:14:45.948
Whereas

251
00:14:47.048 --> 00:14:50.418
<00:14:47.048>high <00:14:47.248>risk <00:14:47.448>are, <00:14:47.648>it <00:14:47.848>basically <00:14:48.048>books <00:14:48.248>you <00:14:48.448>a

252
00:14:50.448 --> 00:14:51.048
table,

253
00:14:52.588 --> 00:14:52.808
paid

254
00:14:53.788 --> 00:14:56.757
<00:14:53.788>a <00:14:54.021>non-refundable <00:14:54.254>deposit <00:14:54.488>of <00:14:54.721>$200 <00:14:54.954>for <00:14:55.188>that

255
00:14:56.788 --> 00:15:00.028
<00:14:56.788>table. <00:14:57.138>That's <00:14:57.488>a <00:14:57.838>high-risk <00:14:58.188>item.

256
00:15:00.188 --> 00:15:04.148
<00:15:00.188>So <00:15:00.363>generally, <00:15:00.538>I <00:15:00.713>would <00:15:00.888>say <00:15:01.063>we <00:15:01.238>should <00:15:01.413>start <00:15:01.588>on

257
00:15:04.208 --> 00:15:07.848
<00:15:04.208>the <00:15:04.348>left, <00:15:04.488>wherein <00:15:04.627>you <00:15:04.767>design <00:15:04.908>your <00:15:05.048>AI <00:15:05.188>features <00:15:05.328>for <00:15:05.468>informing <00:15:05.608>or

258
00:15:07.888 --> 00:15:09.848
<00:15:07.888>suggesting. <00:15:09.288>And

259
00:15:11.228 --> 00:15:15.168
<00:15:11.228>if <00:15:11.383>the <00:15:11.539>actions <00:15:11.694>are <00:15:11.850>reversible, <00:15:12.005>then <00:15:12.161>you <00:15:12.316>probably <00:15:12.472>can <00:15:12.627>have

260
00:15:15.668 --> 00:15:18.988
<00:15:15.668>some <00:15:15.868>kind <00:15:16.068>of <00:15:16.268>advanced <00:15:16.468>feature <00:15:16.668>of <00:15:16.868>act <00:15:17.068>wherein,

261
00:15:19.968 --> 00:15:23.868
<00:15:19.968>if <00:15:20.084>users <00:15:20.201>enable <00:15:20.318>that, <00:15:20.434>it <00:15:20.551>can <00:15:20.668>book <00:15:20.784>tables <00:15:20.901>on <00:15:21.017>their <00:15:21.134>behalf, <00:15:21.251>as <00:15:21.367>long

262
00:15:23.928 --> 00:15:24.688
<00:15:23.928>as <00:15:24.270>it's <00:15:24.612>not

263
00:15:25.568 --> 00:15:28.988
<00:15:25.568>booking <00:15:25.742>the <00:15:25.918>ones <00:15:26.093>that <00:15:26.268>require <00:15:26.443>non-refundable <00:15:26.617>deposit, <00:15:26.793>and <00:15:26.968>it's

264
00:15:29.568 --> 00:15:33.148
<00:15:29.568>reversible. <00:15:29.708>The <00:15:29.848>cost <00:15:29.988>is <00:15:30.127>less <00:15:30.268>over <00:15:30.408>there, <00:15:30.548>so <00:15:30.688>that <00:15:30.828>feature <00:15:30.968>can

265
00:15:33.208 --> 00:15:33.618
exist.

266
00:15:34.888 --> 00:15:38.188
<00:15:34.888>And <00:15:35.015>as <00:15:35.142>I <00:15:35.269>was <00:15:35.397>saying, <00:15:35.524>we <00:15:35.651>should <00:15:35.778>generally <00:15:35.906>start <00:15:36.033>from <00:15:36.160>the <00:15:36.288>left

267
00:15:38.728 --> 00:15:41.968
<00:15:38.728>and <00:15:38.883>slowly <00:15:39.039>go <00:15:39.194>on <00:15:39.350>the <00:15:39.505>right <00:15:39.661>as <00:15:39.816>we <00:15:39.972>see <00:15:40.127>more

268
00:15:42.008 --> 00:15:45.768
<00:15:42.008>examples, <00:15:42.183>more <00:15:42.358>data, <00:15:42.533>and <00:15:42.708>we <00:15:42.883>have <00:15:43.058>more <00:15:43.233>confidence <00:15:43.408>in

269
00:15:45.868 --> 00:15:48.748
<00:15:45.868>our <00:15:46.148>product <00:15:46.428>to <00:15:46.708>go <00:15:46.988>more <00:15:47.268>autonomous.

270
00:15:50.828 --> 00:15:52.928
<00:15:50.828>Failure <00:15:51.108>mode <00:15:51.387>number <00:15:51.668>five, <00:15:51.948>product <00:15:52.228>discipline.

271
00:15:54.968 --> 00:15:58.888
<00:15:54.968>Generally, <00:15:55.108>what <00:15:55.247>happens <00:15:55.387>is <00:15:55.527>we <00:15:55.668>build <00:15:55.808>the <00:15:55.948>product, <00:15:56.088>but <00:15:56.228>we <00:15:56.367>don't

272
00:15:59.028 --> 00:16:01.968
<00:15:59.028>really <00:15:59.228>have <00:15:59.428>much <00:15:59.628>mechanism <00:15:59.828>to <00:16:00.028>get <00:16:00.228>feedback <00:16:00.428>from

273
00:16:02.068 --> 00:16:03.328
users.

274
00:16:04.188 --> 00:16:07.568
<00:16:04.188>And <00:16:04.328>that's <00:16:04.468>something <00:16:04.608>we <00:16:04.747>need <00:16:04.888>to <00:16:05.028>build <00:16:05.168>from <00:16:05.308>the <00:16:05.448>start <00:16:05.588>for.

275
00:16:08.348 --> 00:16:12.268
<00:16:08.348>So <00:16:08.464>we <00:16:08.581>need <00:16:08.698>to <00:16:08.814>instrument <00:16:08.931>everything, <00:16:09.048>wherein <00:16:09.164>we <00:16:09.281>need <00:16:09.397>to <00:16:09.514>collect <00:16:09.631>log <00:16:09.747>prompts,

276
00:16:12.448 --> 00:16:13.908
<00:16:12.448>outputs, <00:16:13.762>confidence

277
00:16:14.808 --> 00:16:15.348
levels,

278
00:16:16.288 --> 00:16:18.788
<00:16:16.288>if <00:16:16.463>the <00:16:16.638>user <00:16:16.813>is <00:16:16.988>overriding, <00:16:17.163>if <00:16:17.338>the <00:16:17.513>user <00:16:17.688>is

279
00:16:19.048 --> 00:16:21.348
<00:16:19.048>rage <00:16:20.448>quitting.

280
00:16:22.468 --> 00:16:22.668
And

281
00:16:23.508 --> 00:16:26.648
<00:16:23.508>from <00:16:23.648>all <00:16:23.788>of <00:16:23.928>that, <00:16:24.068>what <00:16:24.208>we <00:16:24.348>can <00:16:24.488>do <00:16:24.628>is <00:16:24.768>build <00:16:24.908>an

282
00:16:26.688 --> 00:16:30.488
<00:16:26.688>evaluation <00:16:26.888>harness <00:16:27.088>wherein <00:16:27.288>we <00:16:27.488>can <00:16:27.688>have <00:16:27.888>golden <00:16:28.088>sets.

283
00:16:30.528 --> 00:16:33.308
<00:16:30.528>We <00:16:30.683>can <00:16:30.839>use <00:16:30.994>that <00:16:31.150>data <00:16:31.305>for <00:16:31.461>LLM <00:16:31.616>as <00:16:31.772>a <00:16:31.928>judge.

284
00:16:34.748 --> 00:16:38.428
<00:16:34.748>And <00:16:34.948>with <00:16:35.148>that, <00:16:35.348>we <00:16:35.548>close <00:16:35.748>the <00:16:35.948>loop <00:16:36.148>by

285
00:16:38.508 --> 00:16:41.648
<00:16:38.508>using <00:16:38.741>those <00:16:38.974>to <00:16:39.208>improve <00:16:39.441>our <00:16:39.674>next <00:16:39.908>iteration.

286
00:16:42.588 --> 00:16:45.548
<00:16:42.588>And <00:16:42.867>that <00:16:43.147>is <00:16:43.428>really <00:16:43.708>important <00:16:43.988>because

287
00:16:46.368 --> 00:16:50.088
<00:16:46.368>when <00:16:46.508>we <00:16:46.648>create <00:16:46.788>our <00:16:46.928>first <00:16:47.068>set <00:16:47.208>of <00:16:47.348>golden <00:16:47.488>set <00:16:47.628>for <00:16:47.768>evals,

288
00:16:51.208 --> 00:16:54.688
<00:16:51.208>it's <00:16:51.315>always <00:16:51.423>going <00:16:51.531>to <00:16:51.638>be <00:16:51.746>the <00:16:51.854>10 <00:16:51.961>or <00:16:52.069>20 <00:16:52.177>examples <00:16:52.284>we <00:16:52.392>can <00:16:52.500>think <00:16:52.608>about,

289
00:16:54.828 --> 00:16:57.858
<00:16:54.828>whereas <00:16:54.968>in <00:16:55.108>production, <00:16:55.247>the <00:16:55.387>users <00:16:55.528>are <00:16:55.668>going <00:16:55.808>to <00:16:55.948>have <00:16:56.088>so <00:16:56.228>much

290
00:16:58.628 --> 00:17:01.868
<00:16:58.628>valid <00:16:58.783>and <00:16:58.939>nuanced <00:16:59.094>inputs <00:16:59.250>that <00:16:59.405>we <00:16:59.561>didn't <00:16:59.716>even <00:16:59.872>think <00:17:00.028>about.

291
00:17:02.018 --> 00:17:02.338
And

292
00:17:03.488 --> 00:17:06.469
<00:17:03.488>it's <00:17:03.628>the <00:17:03.768>best <00:17:03.908>source <00:17:04.048>for <00:17:04.188>us <00:17:04.328>to <00:17:04.468>be <00:17:04.608>able <00:17:04.748>to <00:17:04.888>expand

293
00:17:07.508 --> 00:17:10.449
<00:17:07.508>our <00:17:07.663>evals <00:17:07.819>as <00:17:07.974>well <00:17:08.130>as <00:17:08.285>improve <00:17:08.441>the <00:17:08.596>product <00:17:08.752>using <00:17:08.908>that.

294
00:17:11.888 --> 00:17:15.209
<00:17:11.887>And <00:17:12.028>they <00:17:12.167>should <00:17:12.308>sit <00:17:12.447>in <00:17:12.588>our <00:17:12.727>CI <00:17:12.867>every <00:17:13.007>time <00:17:13.147>we <00:17:13.288>deploy

295
00:17:15.268 --> 00:17:16.488
something.

296
00:17:17.348 --> 00:17:19.388
<00:17:17.348>It <00:17:17.581>should <00:17:17.814>have <00:17:18.048>LLM <00:17:18.281>as <00:17:18.514>a <00:17:18.748>judge,

297
00:17:20.888 --> 00:17:22.608
<00:17:20.887>which <00:17:21.354>is <00:17:21.821>evaluating <00:17:22.288>or

298
00:17:23.709 --> 00:17:24.748
<00:17:23.709>unit <00:17:24.644>test

299
00:17:25.709 --> 00:17:28.748
<00:17:25.709>evals <00:17:25.989>which <00:17:26.269>run <00:17:26.549>so <00:17:26.829>that <00:17:27.109>every

300
00:17:28.868 --> 00:17:29.948
iteration

301
00:17:31.528 --> 00:17:35.008
<00:17:31.528>of <00:17:31.683>the <00:17:31.839>model <00:17:31.994>of <00:17:32.150>the <00:17:32.305>prompt <00:17:32.461>goes <00:17:32.616>through <00:17:32.772>the <00:17:32.928>same

302
00:17:35.088 --> 00:17:36.948
<00:17:35.088>rigorous <00:17:35.788>process <00:17:36.488>of

303
00:17:38.388 --> 00:17:42.168
<00:17:38.388>a <00:17:38.504>user <00:17:38.621>actually <00:17:38.737>ends <00:17:38.854>up <00:17:38.971>using <00:17:39.088>the <00:17:39.204>product, <00:17:39.321>and <00:17:39.437>we <00:17:39.554>are <00:17:39.671>improving <00:17:39.788>based

304
00:17:42.248 --> 00:17:42.788
<00:17:42.248>on <00:17:42.734>that.

305
00:17:44.368 --> 00:17:47.848
<00:17:44.368>So <00:17:44.475>now <00:17:44.583>coming <00:17:44.691>to <00:17:44.798>the <00:17:44.906>framework <00:17:45.014>that <00:17:45.121>I <00:17:45.229>promised <00:17:45.337>you <00:17:45.444>guys <00:17:45.552>in <00:17:45.660>the <00:17:45.768>start.

306
00:17:50.228 --> 00:17:54.208
<00:17:50.228>It's <00:17:50.368>called <00:17:50.508>TRUST, <00:17:50.648>so <00:17:50.788>it'll <00:17:50.928>not <00:17:51.068>be <00:17:51.208>really <00:17:51.348>hard <00:17:51.488>to <00:17:51.628>remember.

307
00:17:57.168 --> 00:18:00.988
<00:17:57.168>Talking <00:17:57.447>about <00:17:57.727>TRUST, <00:17:58.007>number <00:17:58.287>one, <00:17:58.568>which

308
00:18:01.088 --> 00:18:05.008
<00:18:01.088>is <00:18:01.243>T <00:18:01.399>for <00:18:01.554>task <00:18:01.710>fit. <00:18:01.865>Are <00:18:02.021>you <00:18:02.176>removing <00:18:02.332>steps <00:18:02.488>in

309
00:18:05.048 --> 00:18:07.478
<00:18:05.048>the <00:18:05.328>real <00:18:05.608>workflow? <00:18:05.888>Number <00:18:06.168>two, <00:18:06.448>recovery.

310
00:18:07.648 --> 00:18:10.964
<00:18:07.648>Are <00:18:07.848>there <00:18:08.048>graceful <00:18:08.247>hands <00:18:08.447>off? <00:18:08.648>When <00:18:08.848>it's <00:18:09.048>wrong.

311
00:18:11.684 --> 00:18:14.904
<00:18:11.684>Number <00:18:11.884>three, <00:18:12.084>user <00:18:12.283>control. <00:18:12.484>Is <00:18:12.684>there <00:18:12.884>a <00:18:13.084>right

312
00:18:15.124 --> 00:18:18.924
<00:18:15.124>boundary <00:18:15.279>between <00:18:15.435>suggesting <00:18:15.590>and <00:18:15.746>acting, <00:18:15.901>just <00:18:16.057>like <00:18:16.212>we <00:18:16.368>spoke <00:18:16.524>about

313
00:18:18.964 --> 00:18:22.664
<00:18:18.964>before? <00:18:19.164>Signals, <00:18:19.364>are <00:18:19.563>there <00:18:19.764>feedback <00:18:19.964>loops <00:18:20.164>that <00:18:20.364>drive

314
00:18:22.724 --> 00:18:25.844
<00:18:22.724>improvement? <00:18:23.190>Number <00:18:23.657>five, <00:18:24.124>trust

315
00:18:25.924 --> 00:18:27.964
<00:18:25.924>calibration. <00:18:26.624>Are <00:18:27.324>there

316
00:18:28.784 --> 00:18:31.044
<00:18:28.784>confidence <00:18:29.064>and <00:18:29.344>receipts <00:18:29.624>users <00:18:29.904>can <00:18:30.184>check?

317
00:18:31.604 --> 00:18:35.264
<00:18:31.604>Are <00:18:31.720>you <00:18:31.837>referring <00:18:31.954>to <00:18:32.070>any <00:18:32.187>sources <00:18:32.304>and <00:18:32.420>giving <00:18:32.537>links <00:18:32.654>to <00:18:32.770>that <00:18:32.887>to <00:18:33.004>the

318
00:18:35.424 --> 00:18:35.764
user?

319
00:18:37.284 --> 00:18:37.564
So

320
00:18:38.804 --> 00:18:41.204
<00:18:38.804>what <00:18:38.979>would <00:18:39.154>that <00:18:39.329>actually <00:18:39.504>look <00:18:39.679>like <00:18:39.854>in <00:18:40.029>your <00:18:40.204>stack?

321
00:18:42.844 --> 00:18:44.904
<00:18:42.844>Number <00:18:43.310>one, <00:18:43.777>confidence <00:18:44.244>scaling.

322
00:18:45.764 --> 00:18:48.344
<00:18:45.764>You <00:18:45.997>are <00:18:46.230>computing <00:18:46.464>a <00:18:46.697>score <00:18:46.930>per <00:18:47.164>output,

323
00:18:49.444 --> 00:18:49.744
and

324
00:18:50.704 --> 00:18:53.984
<00:18:50.704>based <00:18:50.859>on <00:18:51.015>that <00:18:51.170>particular <00:18:51.326>score, <00:18:51.481>you <00:18:51.637>are <00:18:51.792>abstaining <00:18:51.948>or <00:18:52.104>downgrading

325
00:18:55.524 --> 00:18:57.244
<00:18:55.524>to <00:18:55.990>suggest <00:18:56.457>when <00:18:56.924>the

326
00:18:58.304 --> 00:18:59.654
<00:18:58.304>score <00:18:58.607>is <00:18:58.911>below <00:18:59.215>the <00:18:59.519>threshold.

327
00:19:00.904 --> 00:19:04.754
<00:19:00.904>Evals <00:19:01.079>in <00:19:01.254>CI, <00:19:01.429>you <00:19:01.604>are <00:19:01.779>creating <00:19:01.954>golden <00:19:02.129>sets <00:19:02.304>and

328
00:19:04.864 --> 00:19:06.464
<00:19:04.864>having <00:19:05.144>LLM <00:19:05.424>as <00:19:05.704>a <00:19:05.984>judge <00:19:06.264>gate

329
00:19:07.684 --> 00:19:11.364
<00:19:07.684>every <00:19:07.839>time <00:19:07.995>you <00:19:08.150>change <00:19:08.306>the <00:19:08.461>prompt, <00:19:08.617>system <00:19:08.772>prompt, <00:19:08.928>or <00:19:09.084>you

330
00:19:11.864 --> 00:19:14.324
<00:19:11.864>change <00:19:12.064>the <00:19:12.264>model. <00:19:12.464>Just <00:19:12.664>like <00:19:12.864>you <00:19:13.064>will <00:19:13.264>have

331
00:19:15.244 --> 00:19:18.924
<00:19:15.244>generally <00:19:15.399>unit <00:19:15.555>tests, <00:19:15.710>integration <00:19:15.866>tests, <00:19:16.021>you <00:19:16.177>are <00:19:16.332>having <00:19:16.488>these <00:19:16.644>new

332
00:19:19.084 --> 00:19:22.804
<00:19:19.084>age <00:19:19.364>evals, <00:19:19.644>for <00:19:19.924>your <00:19:20.204>AI <00:19:20.484>product.

333
00:19:24.204 --> 00:19:26.944
<00:19:24.204>Then <00:19:24.484>having <00:19:24.764>a <00:19:25.043>fallback <00:19:25.323>chain. <00:19:25.604>So

334
00:19:28.784 --> 00:19:32.424
<00:19:28.784>we <00:19:28.924>are <00:19:29.064>building <00:19:29.204>for <00:19:29.344>reliance. <00:19:29.484>So <00:19:29.624>yes, <00:19:29.764>you <00:19:29.904>have <00:19:30.044>a <00:19:30.184>primary

335
00:19:32.484 --> 00:19:35.324
<00:19:32.484>model, <00:19:32.658>but <00:19:32.833>as <00:19:33.009>we <00:19:33.184>all <00:19:33.359>know <00:19:33.533>from <00:19:33.708>our <00:19:33.884>experience,

336
00:19:36.184 --> 00:19:37.704
<00:19:36.184>even <00:19:36.639>the <00:19:37.096>best <00:19:37.552>models,

337
00:19:39.284 --> 00:19:43.184
<00:19:39.284>or <00:19:39.484>even <00:19:39.684>the <00:19:39.884>best <00:19:40.084>providers <00:19:40.284>can <00:19:40.484>have <00:19:40.684>outages.

338
00:19:45.024 --> 00:19:48.284
<00:19:45.024>Talk <00:19:45.198>about <00:19:45.373>infra <00:19:45.549>outages <00:19:45.724>like <00:19:45.899>AWS, <00:19:46.073>or <00:19:46.248>talk <00:19:46.424>about

339
00:19:49.044 --> 00:19:52.844
<00:19:49.044>model <00:19:49.277>capacity <00:19:49.510>outages <00:19:49.744>with <00:19:49.977>Anthropic. <00:19:50.210>It's <00:19:50.444>happening.

340
00:19:52.944 --> 00:19:56.844
<00:19:52.944>It <00:19:53.051>happens <00:19:53.159>all <00:19:53.267>the <00:19:53.374>time. <00:19:53.482>So <00:19:53.590>we <00:19:53.697>need <00:19:53.805>to <00:19:53.913>be <00:19:54.020>prepared <00:19:54.128>so <00:19:54.236>that <00:19:54.344>our

341
00:19:56.904 --> 00:20:00.724
<00:19:56.904>AI <00:19:57.104>product <00:19:57.304>works <00:19:57.504>all <00:19:57.704>the <00:19:57.904>time. <00:19:58.104>So <00:19:58.304>number

342
00:20:00.844 --> 00:20:02.314
<00:20:00.844>one, <00:20:01.174>having <00:20:01.505>a <00:20:01.836>primary <00:20:02.167>model.

343
00:20:03.424 --> 00:20:05.744
<00:20:03.424>Number <00:20:03.657>two, <00:20:03.890>having <00:20:04.124>a <00:20:04.357>cheaper <00:20:04.590>secondary <00:20:04.824>model

344
00:20:06.584 --> 00:20:10.144
<00:20:06.584>that <00:20:06.759>can <00:20:06.934>still <00:20:07.109>reliably <00:20:07.284>give <00:20:07.459>answers <00:20:07.634>to <00:20:07.809>the <00:20:07.984>user.

345
00:20:10.264 --> 00:20:14.214
<00:20:10.264>If <00:20:10.357>that <00:20:10.450>also <00:20:10.543>fails <00:20:10.637>or <00:20:10.730>that's <00:20:10.823>not <00:20:10.917>able <00:20:11.010>to <00:20:11.103>give <00:20:11.197>an <00:20:11.290>answer <00:20:11.383>to <00:20:11.477>the <00:20:11.570>user, <00:20:11.664>then

346
00:20:14.284 --> 00:20:15.864
<00:20:14.284>having <00:20:14.984>cached <00:20:15.684>responses,

347
00:20:16.724 --> 00:20:20.664
<00:20:16.724>answers <00:20:16.957>that <00:20:17.190>have <00:20:17.424>been <00:20:17.657>given <00:20:17.890>before <00:20:18.124>or,

348
00:20:20.744 --> 00:20:24.584
<00:20:20.744>a <00:20:20.899>set <00:20:21.055>of <00:20:21.210>instructions, <00:20:21.366>a <00:20:21.521>set <00:20:21.677>of <00:20:21.832>answers <00:20:21.988>that <00:20:22.144>are

349
00:20:24.604 --> 00:20:28.024
<00:20:24.604>pre-written, <00:20:24.711>which <00:20:24.819>we <00:20:24.927>know <00:20:25.034>that <00:20:25.142>users <00:20:25.250>ask <00:20:25.357>quite <00:20:25.465>a <00:20:25.573>lot, <00:20:25.680>so <00:20:25.788>we <00:20:25.896>are <00:20:26.004>able

350
00:20:28.064 --> 00:20:31.844
<00:20:28.064>to <00:20:28.297>give <00:20:28.530>users <00:20:28.764>a <00:20:28.997>good <00:20:29.230>number <00:20:29.464>of

351
00:20:31.884 --> 00:20:35.844
<00:20:31.884>responses. <00:20:32.039>And <00:20:32.195>number <00:20:32.350>four, <00:20:32.506>if <00:20:32.661>even <00:20:32.817>that's <00:20:32.972>not <00:20:33.128>working, <00:20:33.284>there

352
00:20:35.884 --> 00:20:39.524
<00:20:35.884>should <00:20:36.039>be <00:20:36.195>humans <00:20:36.350>who <00:20:36.506>should <00:20:36.661>be <00:20:36.817>responding <00:20:36.972>to <00:20:37.128>the <00:20:37.284>users.

353
00:20:40.604 --> 00:20:41.924
<00:20:40.604>We <00:20:40.802>should <00:20:41.000>never <00:20:41.198>ship <00:20:41.396>a <00:20:41.594>dead <00:20:41.792>end.

354
00:20:43.164 --> 00:20:45.384
<00:20:43.164>Then <00:20:43.444>coming <00:20:43.724>to <00:20:44.004>grounding <00:20:44.283>and <00:20:44.564>citations.

355
00:20:46.604 --> 00:20:50.564
<00:20:46.604>So <00:20:46.804>providing <00:20:47.004>tool <00:20:47.204>call <00:20:47.404>traces, <00:20:47.604>RAG <00:20:47.804>context, <00:20:48.004>and

356
00:20:51.744 --> 00:20:55.484
<00:20:51.744>basically <00:20:51.851>making <00:20:51.959>each <00:20:52.067>line <00:20:52.174>that <00:20:52.282>we <00:20:52.390>are <00:20:52.497>writing, <00:20:52.605>especially <00:20:52.713>when <00:20:52.820>we <00:20:52.928>are <00:20:53.036>referring <00:20:53.144>to

357
00:20:55.524 --> 00:20:57.524
<00:20:55.524>something <00:20:55.873>and <00:20:56.224>providing <00:20:56.573>important <00:20:56.924>information,

358
00:20:58.764 --> 00:21:02.554
<00:20:58.764>verifiable <00:20:58.871>for <00:20:58.979>the <00:20:59.087>user <00:20:59.194>that <00:20:59.302>they <00:20:59.410>can <00:20:59.517>go <00:20:59.625>and <00:20:59.733>read <00:20:59.840>in <00:20:59.948>the <00:21:00.056>link <00:21:00.164>themselves,

359
00:21:04.364 --> 00:21:06.684
<00:21:04.364>so <00:21:04.830>that <00:21:05.297>it's <00:21:05.764>basically,

360
00:21:08.684 --> 00:21:12.224
<00:21:08.684>we <00:21:08.859>are <00:21:09.033>reducing <00:21:09.209>the <00:21:09.384>risk <00:21:09.559>of <00:21:09.734>the <00:21:09.908>user <00:21:10.084>perceiving

361
00:21:12.864 --> 00:21:14.704
<00:21:12.864>the <00:21:13.144>answer <00:21:13.424>as <00:21:13.704>a <00:21:13.984>hallucinated <00:21:14.264>answer.

362
00:21:15.684 --> 00:21:19.504
<00:21:15.684>Coming <00:21:16.033>then <00:21:16.384>to <00:21:16.734>observability, <00:21:17.084>which

363
00:21:19.564 --> 00:21:22.764
<00:21:19.564>is <00:21:19.797>where <00:21:20.030>we <00:21:20.264>can <00:21:20.497>trace <00:21:20.730>prompts, <00:21:20.964>outputs,

364
00:21:22.864 --> 00:21:25.904
<00:21:22.864>confidence, <00:21:23.097>latency, <00:21:23.330>and <00:21:23.564>user <00:21:23.797>action <00:21:24.030>end <00:21:24.264>to

365
00:21:26.004 --> 00:21:29.864
<00:21:26.004>end. <00:21:26.237>And <00:21:26.470>then, <00:21:26.704>staged <00:21:26.937>autonomy. <00:21:27.170>So <00:21:27.404>you

366
00:21:30.004 --> 00:21:31.724
<00:21:30.004>can <00:21:30.353>have <00:21:30.704>an <00:21:31.053>act <00:21:31.404>part,

367
00:21:32.604 --> 00:21:36.404
<00:21:32.604>but <00:21:32.720>you <00:21:32.837>don't <00:21:32.954>launch <00:21:33.070>it <00:21:33.187>until <00:21:33.304>you <00:21:33.420>have <00:21:33.537>enough <00:21:33.654>data <00:21:33.770>to <00:21:33.887>prove <00:21:34.004>that,

368
00:21:36.524 --> 00:21:40.224
<00:21:36.524>yes, <00:21:36.664>the <00:21:36.803>model <00:21:36.944>is <00:21:37.083>capable. <00:21:37.224>Maybe <00:21:37.363>you <00:21:37.504>release <00:21:37.643>it <00:21:37.783>to <00:21:37.924>a

369
00:21:40.284 --> 00:21:44.234
<00:21:40.284>certain <00:21:40.484>section <00:21:40.684>of <00:21:40.884>the <00:21:41.084>users <00:21:41.284>who <00:21:41.484>sign <00:21:41.684>into

370
00:21:44.244 --> 00:21:45.884
<00:21:44.244>the <00:21:44.593>risks <00:21:44.944>that <00:21:45.293>are <00:21:45.644>there.

371
00:21:48.024 --> 00:21:51.504
<00:21:48.024>So <00:21:48.117>now <00:21:48.210>we're <00:21:48.303>going <00:21:48.397>to <00:21:48.490>talk <00:21:48.583>about <00:21:48.677>how <00:21:48.770>does <00:21:48.863>it <00:21:48.957>look <00:21:49.050>in <00:21:49.143>practice, <00:21:49.237>the <00:21:49.330>framework, <00:21:49.424>the

372
00:21:51.584 --> 00:21:54.444
<00:21:51.584>real <00:21:51.759>code, <00:21:51.934>evals, <00:21:52.109>the <00:21:52.284>agents <00:21:52.459>that <00:21:52.634>we <00:21:52.809>spoke <00:21:52.984>about.

373
00:21:56.164 --> 00:21:57.964
<00:21:56.164>Number <00:21:56.444>one, <00:21:56.724>coming <00:21:57.004>to <00:21:57.283>confidence <00:21:57.564>gating.

374
00:21:58.684 --> 00:21:58.984
So

375
00:22:00.504 --> 00:22:02.864
<00:22:00.504>it's <00:22:00.704>a <00:22:00.904>simple <00:22:01.103>code <00:22:01.303>like <00:22:01.504>this <00:22:01.704>that <00:22:01.904>can

376
00:22:04.244 --> 00:22:07.804
<00:22:04.244>have <00:22:04.351>an <00:22:04.459>if <00:22:04.567>else <00:22:04.674>block <00:22:04.782>based <00:22:04.890>on <00:22:04.997>the <00:22:05.105>confidence <00:22:05.213>score, <00:22:05.320>and <00:22:05.428>we <00:22:05.536>get <00:22:05.644>the

377
00:22:07.864 --> 00:22:10.364
<00:22:07.864>confidence <00:22:08.064>score <00:22:08.264>from <00:22:08.464>the <00:22:08.664>log <00:22:08.864>probes <00:22:09.064>or <00:22:09.264>verifiers.

378
00:22:11.164 --> 00:22:14.344
<00:22:11.164>And <00:22:11.339>based <00:22:11.514>on <00:22:11.689>the <00:22:11.864>confidence <00:22:12.039>score <00:22:12.214>we <00:22:12.389>are <00:22:12.564>receiving

379
00:22:14.984 --> 00:22:18.164
<00:22:14.984>from <00:22:15.158>that, <00:22:15.333>we <00:22:15.509>either <00:22:15.684>ask <00:22:15.859>the <00:22:16.033>agent <00:22:16.208>to <00:22:16.384>act

380
00:22:18.544 --> 00:22:21.684
<00:22:18.544>or <00:22:18.777>if <00:22:19.010>the <00:22:19.244>confidence <00:22:19.477>is <00:22:19.710>lower, <00:22:19.944>suggest.

381
00:22:22.344 --> 00:22:23.334
<00:22:22.344>If <00:22:22.641>it's <00:22:22.938>even <00:22:23.235>lower,

382
00:22:25.284 --> 00:22:29.064
<00:22:25.284>probably <00:22:25.411>go <00:22:25.538>for <00:22:25.665>a <00:22:25.793>cached <00:22:25.920>answer <00:22:26.047>or <00:22:26.174>hand <00:22:26.302>off <00:22:26.429>to <00:22:26.556>a <00:22:26.684>human.

383
00:22:29.804 --> 00:22:33.744
<00:22:29.804>So <00:22:29.891>those <00:22:29.979>are <00:22:30.066>some <00:22:30.154>of <00:22:30.241>the <00:22:30.329>things <00:22:30.416>that <00:22:30.504>we <00:22:30.591>can <00:22:30.679>do <00:22:30.766>based <00:22:30.854>on <00:22:30.941>the <00:22:31.029>confidence <00:22:31.116>score <00:22:31.204>that

384
00:22:33.824 --> 00:22:35.264
<00:22:33.824>we <00:22:35.120>assess.

385
00:22:36.244 --> 00:22:36.504
Then

386
00:22:37.424 --> 00:22:40.984
<00:22:37.424>treating <00:22:37.624>prompts <00:22:37.824>like <00:22:38.024>code, <00:22:38.224>creating <00:22:38.424>a <00:22:38.624>golden <00:22:38.824>set.

387
00:22:41.644 --> 00:22:45.604
<00:22:41.644>So <00:22:41.784>what <00:22:41.924>does <00:22:42.064>a <00:22:42.204>set <00:22:42.344>look <00:22:42.484>like? <00:22:42.624>You <00:22:42.764>have <00:22:42.904>an <00:22:43.044>input,

388
00:22:45.744 --> 00:22:49.664
<00:22:45.744>you <00:22:45.860>have <00:22:45.977>an <00:22:46.093>expected <00:22:46.210>behavior, <00:22:46.327>and <00:22:46.444>you <00:22:46.560>have <00:22:46.677>an <00:22:46.793>automated <00:22:46.910>check <00:22:47.027>that <00:22:47.144>gets

389
00:22:49.744 --> 00:22:51.004
<00:22:49.744>done <00:22:50.027>on <00:22:50.311>each <00:22:50.594>of <00:22:50.878>these.

390
00:22:52.244 --> 00:22:55.344
<00:22:52.244>And <00:22:52.384>I <00:22:52.524>think <00:22:52.664>another <00:22:52.803>interesting <00:22:52.944>concept <00:22:53.083>is <00:22:53.224>LLM <00:22:53.363>as <00:22:53.504>a <00:22:53.644>judge,

391
00:22:55.804 --> 00:22:59.704
<00:22:55.804>wherein <00:22:55.931>on <00:22:56.058>all <00:22:56.185>of <00:22:56.313>these, <00:22:56.440>you <00:22:56.567>run <00:22:56.694>a <00:22:56.822>LLM <00:22:56.949>as <00:22:57.076>a <00:22:57.204>judge,

392
00:22:59.844 --> 00:23:01.204
<00:22:59.844>which <00:23:00.252>is <00:23:00.660>like <00:23:01.068>your

393
00:23:02.244 --> 00:23:05.624
<00:23:02.244>invigilator <00:23:02.944>or <00:23:03.644>an

394
00:23:05.684 --> 00:23:08.754
<00:23:05.684>examiner <00:23:05.884>checking <00:23:06.084>your <00:23:06.283>exam <00:23:06.484>sheet, <00:23:06.684>answer <00:23:06.884>sheet <00:23:07.084>after,

395
00:23:09.364 --> 00:23:12.664
<00:23:09.364>wherein <00:23:09.539>it <00:23:09.714>checks <00:23:09.889>whether <00:23:10.064>all <00:23:10.239>the <00:23:10.414>answers <00:23:10.589>were <00:23:10.764>grounded,

396
00:23:13.004 --> 00:23:15.004
<00:23:13.004>whether <00:23:13.353>real <00:23:13.704>sources <00:23:14.053>were <00:23:14.404>cited,

397
00:23:16.864 --> 00:23:20.704
<00:23:16.864>did <00:23:17.019>it <00:23:17.175>classify <00:23:17.330>the <00:23:17.486>intent <00:23:17.641>correctly, <00:23:17.797>was <00:23:17.952>it <00:23:18.108>safe, <00:23:18.264>was

398
00:23:20.764 --> 00:23:24.544
<00:23:20.764>it <00:23:20.904>calibrated? <00:23:21.043>And <00:23:21.184>it <00:23:21.323>gives <00:23:21.464>a <00:23:21.603>score <00:23:21.744>based <00:23:21.883>on <00:23:22.024>that, <00:23:22.164>and

399
00:23:24.584 --> 00:23:28.304
<00:23:24.584>then <00:23:24.739>passes <00:23:24.895>your <00:23:25.050>CI <00:23:25.206>or <00:23:25.361>fails <00:23:25.517>your <00:23:25.672>CI <00:23:25.828>based <00:23:25.984>on

400
00:23:28.364 --> 00:23:32.284
<00:23:28.364>that. <00:23:28.714>So, <00:23:29.064>having <00:23:29.414>an <00:23:29.764>LLM

401
00:23:32.364 --> 00:23:36.284
<00:23:32.364>as <00:23:32.597>a <00:23:32.830>kind <00:23:33.064>of <00:23:33.297>teacher/judge <00:23:33.530>who <00:23:33.764>basically

402
00:23:36.364 --> 00:23:39.424
<00:23:36.364>marks <00:23:36.519>your <00:23:36.675>mark <00:23:36.830>sheet, <00:23:36.986>and <00:23:37.141>gives <00:23:37.297>a <00:23:37.452>mark <00:23:37.608>sheet <00:23:37.764>is

403
00:23:40.224 --> 00:23:41.604
<00:23:40.224>really <00:23:40.638>valuable <00:23:41.052>over <00:23:41.466>here.

404
00:23:43.644 --> 00:23:45.704
<00:23:43.644>Now <00:23:43.844>I <00:23:44.044>want <00:23:44.244>to <00:23:44.444>give <00:23:44.644>an <00:23:44.844>example <00:23:45.044>of

405
00:23:47.224 --> 00:23:48.764
<00:23:47.224>a <00:23:47.917>real-life <00:23:48.610>production

406
00:23:49.564 --> 00:23:53.264
<00:23:49.564>tool <00:23:49.764>that <00:23:49.964>I've <00:23:50.164>built <00:23:50.364>for <00:23:50.564>our <00:23:50.764>organization <00:23:50.964>to

407
00:23:53.404 --> 00:23:53.684
use,

408
00:23:54.534 --> 00:23:57.604
<00:23:54.534>and <00:23:54.814>it <00:23:55.094>sustained <00:23:55.374>the <00:23:55.654>test <00:23:55.934>adoption.

409
00:23:59.684 --> 00:24:03.544
<00:23:59.684>So <00:23:59.884>how <00:24:00.084>does <00:24:00.283>it <00:24:00.484>work? <00:24:00.684>All <00:24:00.884>our <00:24:01.084>users

410
00:24:03.564 --> 00:24:06.844
<00:24:03.564>sit <00:24:03.797>on <00:24:04.030>Slack. <00:24:04.264>So <00:24:04.497>I <00:24:04.730>built <00:24:04.964>this

411
00:24:07.304 --> 00:24:09.364
<00:24:07.304>communication <00:24:07.770>interface <00:24:08.237>on <00:24:08.704>Slack.

412
00:24:11.144 --> 00:24:14.384
<00:24:11.144>So <00:24:11.319>what <00:24:11.494>does <00:24:11.669>it <00:24:11.844>do? <00:24:12.019>Basically, <00:24:12.194>a <00:24:12.369>user <00:24:12.544>types

413
00:24:14.524 --> 00:24:17.204
<00:24:14.524>@agent <00:24:14.757>build <00:24:14.990>me <00:24:15.224>an <00:24:15.457>automation <00:24:15.690>for <00:24:15.924>this

414
00:24:18.564 --> 00:24:22.124
<00:24:18.564>Then <00:24:18.719>I've <00:24:18.875>got <00:24:19.030>a <00:24:19.186>communication <00:24:19.341>gating <00:24:19.497>between <00:24:19.652>Slack <00:24:19.808>and <00:24:19.964>OpenClau,

415
00:24:22.904 --> 00:24:24.204
<00:24:22.904>wherein <00:24:24.074>this

416
00:24:26.204 --> 00:24:30.074
<00:24:26.204>application <00:24:26.484>basically <00:24:26.764>uses <00:24:27.043>GPT-4o <00:24:27.323>mini <00:24:27.604>and

417
00:24:30.224 --> 00:24:33.444
<00:24:30.224>assesses <00:24:30.364>the <00:24:30.504>intent <00:24:30.644>of <00:24:30.783>the <00:24:30.924>user, <00:24:31.063>whether <00:24:31.204>there's <00:24:31.343>a <00:24:31.484>destructive <00:24:31.624>intent,

418
00:24:34.284 --> 00:24:38.104
<00:24:34.284>blocks <00:24:34.400>the <00:24:34.517>request <00:24:34.634>in <00:24:34.750>those <00:24:34.867>cases, <00:24:34.984>and <00:24:35.100>if <00:24:35.217>the <00:24:35.334>user <00:24:35.450>is <00:24:35.567>authorized <00:24:35.684>to

419
00:24:38.144 --> 00:24:41.464
<00:24:38.144>make <00:24:38.377>this <00:24:38.610>task, <00:24:38.844>et <00:24:39.077>cetera. <00:24:39.310>And <00:24:39.544>then

420
00:24:43.444 --> 00:24:46.484
<00:24:43.444>once <00:24:43.644>it's <00:24:43.844>done <00:24:44.043>that, <00:24:44.244>it <00:24:44.444>basically <00:24:44.644>just <00:24:44.844>becomes

421
00:24:47.444 --> 00:24:51.223
<00:24:47.444>a <00:24:47.644>communication <00:24:47.844>tool <00:24:48.043>in <00:24:48.244>between <00:24:48.444>Slack <00:24:48.644>and <00:24:48.844>OpenClau.

422
00:24:52.324 --> 00:24:54.324
<00:24:52.324>Then <00:24:52.557>it <00:24:52.790>passes <00:24:53.024>the <00:24:53.257>information <00:24:53.490>to <00:24:53.724>OpenClau.

423
00:24:55.424 --> 00:24:58.824
<00:24:55.424>And <00:24:55.624>now <00:24:55.824>over <00:24:56.024>here, <00:24:56.224>I <00:24:56.424>have <00:24:56.624>got <00:24:56.824>an

424
00:24:58.864 --> 00:25:01.404
<00:24:58.864>automation <00:24:59.330>agent <00:24:59.797>within <00:25:00.264>OpenClau

425
00:25:02.684 --> 00:25:06.564
<00:25:02.684>that <00:25:02.839>has <00:25:02.995>context <00:25:03.150>on <00:25:03.306>the <00:25:03.461>organizations <00:25:03.617>or <00:25:03.772>the <00:25:03.928>products <00:25:04.084>we

426
00:25:06.624 --> 00:25:08.364
<00:25:06.624>build, <00:25:06.824>the <00:25:07.024>kind <00:25:07.224>of <00:25:07.424>work <00:25:07.624>that <00:25:07.824>we <00:25:08.024>do.

427
00:25:09.473 --> 00:25:13.144
<00:25:09.473>And <00:25:09.628>we've <00:25:09.784>got <00:25:09.939>one <00:25:10.095>of <00:25:10.250>the <00:25:10.406>capable <00:25:10.561>models <00:25:10.717>like <00:25:10.873>Opus

428
00:25:13.284 --> 00:25:14.664
<00:25:13.284>4.8 <00:25:13.698>over <00:25:14.112>here, <00:25:14.526>which

429
00:25:16.564 --> 00:25:20.484
<00:25:16.564>tries <00:25:16.680>to <00:25:16.797>understand <00:25:16.914>what <00:25:17.030>the <00:25:17.147>user <00:25:17.264>wants <00:25:17.380>and <00:25:17.497>goes <00:25:17.614>back <00:25:17.730>and <00:25:17.847>forth <00:25:17.964>with

430
00:25:20.544 --> 00:25:23.664
<00:25:20.544>this <00:25:20.719>WebSocket <00:25:20.894>connection <00:25:21.069>tool <00:25:21.244>that <00:25:21.419>we've <00:25:21.594>built <00:25:21.769>in <00:25:21.944>between,

431
00:25:25.164 --> 00:25:26.764
<00:25:25.164>goes <00:25:25.397>back <00:25:25.630>to <00:25:25.864>the <00:25:26.097>user <00:25:26.330>on <00:25:26.564>Slack,

432
00:25:27.564 --> 00:25:31.444
<00:25:27.564>confirms <00:25:27.704>all <00:25:27.844>the <00:25:27.984>business <00:25:28.124>context, <00:25:28.264>any <00:25:28.404>ambiguity <00:25:28.544>that <00:25:28.684>it <00:25:28.824>has, <00:25:28.964>it

433
00:25:31.504 --> 00:25:34.524
<00:25:31.504>tries <00:25:31.644>to <00:25:31.783>resolve <00:25:31.924>with <00:25:32.063>the <00:25:32.204>user <00:25:32.343>in <00:25:32.484>multiple <00:25:32.623>back <00:25:32.764>and <00:25:32.904>forth.

434
00:25:35.784 --> 00:25:36.534
<00:25:35.784>Now, <00:25:36.459>once

435
00:25:37.724 --> 00:25:39.644
<00:25:37.724>it's <00:25:37.957>basically <00:25:38.190>understood <00:25:38.424>everything, <00:25:38.657>it <00:25:38.890>goes <00:25:39.124>and

436
00:25:40.804 --> 00:25:42.904
<00:25:40.804>researches <00:25:41.154>online, <00:25:41.504>creates <00:25:41.854>a <00:25:42.204>plan.

437
00:25:43.424 --> 00:25:45.724
<00:25:43.424>For <00:25:43.657>example, <00:25:43.890>I <00:25:44.124>want <00:25:44.357>all <00:25:44.590>the <00:25:44.824>news

438
00:25:46.664 --> 00:25:49.544
<00:25:46.664>about <00:25:47.014>these <00:25:47.364>companies <00:25:47.714>every <00:25:48.064>morning.

439
00:25:50.364 --> 00:25:53.984
<00:25:50.364>It <00:25:50.539>will <00:25:50.714>probably <00:25:50.889>create <00:25:51.064>a <00:25:51.239>plan <00:25:51.414>wherein <00:25:51.589>it'll <00:25:51.764>fetch

440
00:25:54.204 --> 00:25:57.634
<00:25:54.204>Google <00:25:54.553>RSS <00:25:54.904>feeds. <00:25:55.254>It'll <00:25:55.604>probably

441
00:25:58.864 --> 00:26:02.604
<00:25:58.864>identify <00:25:58.991>some <00:25:59.118>websites <00:25:59.245>which <00:25:59.373>have <00:25:59.500>latest <00:25:59.627>news <00:25:59.754>on <00:25:59.882>these <00:26:00.009>or <00:26:00.136>information <00:26:00.264>on

442
00:26:02.664 --> 00:26:06.284
<00:26:02.664>these. <00:26:02.791>Now, <00:26:02.918>once <00:26:03.045>it's <00:26:03.173>done <00:26:03.300>that, <00:26:03.427>it'll <00:26:03.554>come <00:26:03.682>back <00:26:03.809>to <00:26:03.936>the <00:26:04.064>user,

443
00:26:06.864 --> 00:26:08.264
<00:26:06.864>give <00:26:07.179>them <00:26:07.494>an <00:26:07.809>approved <00:26:08.123>plan.

444
00:26:10.744 --> 00:26:14.364
<00:26:10.744>If <00:26:10.851>the <00:26:10.959>user <00:26:11.067>agrees, <00:26:11.174>it's <00:26:11.282>going <00:26:11.390>to <00:26:11.497>go <00:26:11.605>and <00:26:11.713>test <00:26:11.820>the <00:26:11.928>plan, <00:26:12.036>create <00:26:12.144>a

445
00:26:14.404 --> 00:26:17.764
<00:26:14.404>script <00:26:14.604>that <00:26:14.804>works. <00:26:15.004>Once <00:26:15.204>it's <00:26:15.404>got <00:26:15.604>the <00:26:15.804>script,

446
00:26:18.344 --> 00:26:22.164
<00:26:18.344>it's <00:26:18.499>done <00:26:18.655>the <00:26:18.810>test, <00:26:18.966>it's <00:26:19.121>working, <00:26:19.277>it's <00:26:19.432>assessed <00:26:19.588>the <00:26:19.744>results,

447
00:26:22.204 --> 00:26:25.484
<00:26:22.204>it <00:26:22.404>goes <00:26:22.604>and <00:26:22.803>creates <00:26:23.004>a <00:26:23.204>script <00:26:23.404>in <00:26:23.604>Winmill.

448
00:26:25.584 --> 00:26:28.284
<00:26:25.584>Winmill <00:26:25.864>is <00:26:26.144>an <00:26:26.424>automation <00:26:26.704>tool <00:26:26.984>which

449
00:26:29.164 --> 00:26:31.044
<00:26:29.164>is <00:26:29.630>more <00:26:30.097>script <00:26:30.564>based.

450
00:26:32.414 --> 00:26:32.984
<00:26:32.414>And <00:26:32.927>then

451
00:26:34.944 --> 00:26:38.933
<00:26:34.944>Winmill <00:26:35.060>schedules <00:26:35.177>the <00:26:35.293>job. <00:26:35.410>For <00:26:35.527>example, <00:26:35.644>I <00:26:35.760>need <00:26:35.877>it <00:26:35.994>every <00:26:36.110>morning <00:26:36.227>at <00:26:36.344>9:00

452
00:26:38.964 --> 00:26:41.464
<00:26:38.964>a.m., <00:26:39.104>so <00:26:39.244>Winmill <00:26:39.384>is <00:26:39.524>going <00:26:39.664>to <00:26:39.803>run <00:26:39.944>that <00:26:40.083>at <00:26:40.224>9:00 <00:26:40.364>a.m.

453
00:26:42.824 --> 00:26:45.064
<00:26:42.824>And <00:26:43.024>to <00:26:43.224>run <00:26:43.424>it, <00:26:43.624>we <00:26:43.824>use <00:26:44.024>AWS <00:26:44.224>Fargate,

454
00:26:46.344 --> 00:26:49.584
<00:26:46.344>which <00:26:46.499>is <00:26:46.655>where <00:26:46.810>our <00:26:46.966>containers <00:26:47.121>spawn <00:26:47.277>up, <00:26:47.432>and <00:26:47.588>based <00:26:47.744>on

455
00:26:50.004 --> 00:26:53.684
<00:26:50.004>if <00:26:50.470>we <00:26:50.937>have <00:26:51.404>10

456
00:26:53.704 --> 00:26:57.484
<00:26:53.704>automations <00:26:53.811>at <00:26:53.919>9:00 <00:26:54.027>a.m., <00:26:54.134>then <00:26:54.242>there'll <00:26:54.350>be <00:26:54.457>10 <00:26:54.565>workers, <00:26:54.673>and <00:26:54.780>if <00:26:54.888>there <00:26:54.996>are <00:26:55.104>1,000

457
00:26:57.524 --> 00:27:01.144
<00:26:57.524>at <00:26:57.664>12:00 <00:26:57.803>p.m., <00:26:57.944>then <00:26:58.083>there'll <00:26:58.224>be <00:26:58.363>1,000 <00:26:58.504>workers <00:26:58.643>at <00:26:58.783>12:00 <00:26:58.924>p.m.

458
00:27:01.384 --> 00:27:04.004
<00:27:01.384>So <00:27:01.734>it's <00:27:02.084>very <00:27:02.434>scalable, <00:27:02.784>containerized.

459
00:27:05.384 --> 00:27:05.704
It

460
00:27:07.404 --> 00:27:11.164
<00:27:07.404>basically <00:27:07.870>can <00:27:08.337>do <00:27:08.804>pretty

461
00:27:11.204 --> 00:27:14.314
<00:27:11.204>much <00:27:11.379>what <00:27:11.553>you <00:27:11.729>want <00:27:11.904>it <00:27:12.079>to <00:27:12.254>because <00:27:12.428>everything's <00:27:12.604>script

462
00:27:14.324 --> 00:27:17.884
<00:27:14.324>based. <00:27:14.524>It's <00:27:14.724>got <00:27:14.924>some <00:27:15.124>limitations <00:27:15.324>that <00:27:15.524>we <00:27:15.724>have

463
00:27:18.004 --> 00:27:21.344
<00:27:18.004>created <00:27:18.237>so <00:27:18.470>that <00:27:18.704>it <00:27:18.937>does <00:27:19.170>not, <00:27:19.404>for

464
00:27:21.404 --> 00:27:25.124
<00:27:21.404>example, <00:27:21.637>go <00:27:21.870>overboard <00:27:22.104>on <00:27:22.337>the <00:27:22.570>cost, <00:27:22.804>like

465
00:27:25.504 --> 00:27:27.964
<00:27:25.504>using <00:27:25.737>some <00:27:25.970>API <00:27:26.204>which <00:27:26.437>is <00:27:26.670>very <00:27:26.904>expensive.

466
00:27:28.584 --> 00:27:30.364
<00:27:28.584>Of <00:27:28.817>course, <00:27:29.050>it <00:27:29.284>needs <00:27:29.517>the <00:27:29.750>keys <00:27:29.984>or

467
00:27:32.904 --> 00:27:36.713
<00:27:32.904>basically <00:27:33.104>doing <00:27:33.304>anything <00:27:33.504>like <00:27:33.704>working <00:27:33.904>with <00:27:34.104>transactions, <00:27:34.304>et

468
00:27:36.713 --> 00:27:37.124
cetera.

469
00:27:38.384 --> 00:27:42.084
<00:27:38.384>But <00:27:38.477>at <00:27:38.570>the <00:27:38.664>end <00:27:38.757>of <00:27:38.850>the <00:27:38.944>day, <00:27:39.037>nothing <00:27:39.130>gets <00:27:39.224>built <00:27:39.317>or <00:27:39.410>run <00:27:39.504>until <00:27:39.597>you <00:27:39.690>confirm <00:27:39.784>the

470
00:27:42.124 --> 00:27:45.394
<00:27:42.124>intent <00:27:42.279>and <00:27:42.435>plan <00:27:42.590>in <00:27:42.746>Slack. <00:27:42.901>So <00:27:43.057>we've <00:27:43.212>created <00:27:43.368>that <00:27:43.524>guard

471
00:27:45.784 --> 00:27:48.944
<00:27:45.784>rail <00:27:46.484>or <00:27:47.184>railing.

472
00:27:50.894 --> 00:27:53.544
<00:27:50.894>And <00:27:51.049>whatever <00:27:51.205>the <00:27:51.360>output <00:27:51.516>is, <00:27:51.671>it <00:27:51.827>goes <00:27:51.982>back <00:27:52.138>to <00:27:52.294>Slack.

473
00:27:53.604 --> 00:27:56.964
<00:27:53.604>So, <00:27:54.304>this <00:27:55.004>is

474
00:27:57.004 --> 00:27:59.044
<00:27:57.004>basically <00:27:57.353>what <00:27:57.704>we <00:27:58.053>have <00:27:58.404>created.

475
00:28:01.384 --> 00:28:04.204
<00:28:01.384>In <00:28:01.584>general, <00:28:01.784>in <00:28:01.984>our <00:28:02.184>demos, <00:28:02.384>everything <00:28:02.584>just <00:28:02.784>runs.

476
00:28:04.384 --> 00:28:07.294
<00:28:04.384>Over <00:28:04.539>here <00:28:04.695>in <00:28:04.850>production, <00:28:05.006>we <00:28:05.161>are <00:28:05.317>confirming <00:28:05.472>the <00:28:05.628>plan <00:28:05.784>first.

477
00:28:07.294 --> 00:28:08.284
<00:28:07.294>So <00:28:07.591>that's <00:28:07.888>the <00:28:08.185>difference.

478
00:28:09.844 --> 00:28:13.684
<00:28:09.844>So <00:28:09.984>some <00:28:10.124>of <00:28:10.264>the <00:28:10.404>recommendation, <00:28:10.544>confirm <00:28:10.684>intent <00:28:10.824>and <00:28:10.964>plan, <00:28:11.104>test <00:28:11.244>before

479
00:28:13.784 --> 00:28:16.824
<00:28:13.784>you <00:28:13.984>start <00:28:14.184>building, <00:28:14.384>discovery <00:28:14.584>that <00:28:14.784>we <00:28:14.984>generally <00:28:15.184>do.

480
00:28:17.764 --> 00:28:21.304
<00:28:17.764>And <00:28:17.880>over <00:28:17.997>here, <00:28:18.113>we <00:28:18.230>have <00:28:18.347>created <00:28:18.464>one <00:28:18.580>channel <00:28:18.697>which <00:28:18.813>is <00:28:18.930>in <00:28:19.047>and <00:28:19.164>out

481
00:28:21.384 --> 00:28:22.664
<00:28:21.384>basically <00:28:21.960>with <00:28:22.536>Slack.

482
00:28:23.504 --> 00:28:27.204
<00:28:23.504>The <00:28:23.611>communication <00:28:23.719>is <00:28:23.827>coming <00:28:23.934>over <00:28:24.042>here <00:28:24.150>and <00:28:24.257>going <00:28:24.365>out <00:28:24.473>from <00:28:24.580>here <00:28:24.688>to <00:28:24.796>the <00:28:24.904>agent.

483
00:28:31.844 --> 00:28:35.473
<00:28:31.844>Now <00:28:31.960>coming <00:28:32.077>to <00:28:32.194>the <00:28:32.310>most <00:28:32.427>exciting <00:28:32.544>part, <00:28:32.660>which <00:28:32.777>is <00:28:32.894>part <00:28:33.010>five, <00:28:33.127>ship <00:28:33.244>it.

484
00:28:37.124 --> 00:28:39.334
<00:28:37.124>First, <00:28:37.404>having <00:28:37.684>a <00:28:37.964>pre-ship <00:28:38.244>trust <00:28:38.524>review,

485
00:28:40.184 --> 00:28:43.144
<00:28:40.184>which <00:28:40.384>as <00:28:40.584>we <00:28:40.783>discussed <00:28:40.984>using <00:28:41.184>the <00:28:41.384>framework <00:28:41.584>TRUST.

486
00:28:43.624 --> 00:28:47.164
<00:28:43.624>T <00:28:44.090>for <00:28:44.557>task <00:28:45.024>fit,

487
00:28:47.424 --> 00:28:51.314
<00:28:47.424>which <00:28:47.579>is, <00:28:47.735>does <00:28:47.890>this <00:28:48.046>remove <00:28:48.201>a <00:28:48.357>step <00:28:48.512>users <00:28:48.668>hate <00:28:48.824>or

488
00:28:51.384 --> 00:28:54.424
<00:28:51.384>add <00:28:51.559>a <00:28:51.734>new <00:28:51.909>destination <00:28:52.084>or <00:28:52.259>increase <00:28:52.434>the <00:28:52.609>number <00:28:52.784>of

489
00:28:54.484 --> 00:28:58.464
<00:28:54.484>steps? <00:28:54.684>Recovery, <00:28:54.884>what's <00:28:55.083>the <00:28:55.283>experience <00:28:55.484>when <00:28:55.684>the <00:28:55.884>model

490
00:28:58.524 --> 00:29:01.264
<00:28:58.524>is <00:28:58.724>unsure, <00:28:58.924>ambiguous, <00:28:59.123>or <00:28:59.323>wrong <00:28:59.524>for <00:28:59.724>the <00:28:59.924>user?

491
00:29:02.424 --> 00:29:06.044
<00:29:02.424>Number <00:29:02.599>three, <00:29:02.774>user <00:29:02.949>control. <00:29:03.124>Where <00:29:03.299>on <00:29:03.474>that <00:29:03.649>inform <00:29:03.824>or

492
00:29:06.064 --> 00:29:10.024
<00:29:06.064>suggest <00:29:06.171>to <00:29:06.279>act <00:29:06.387>spectrum <00:29:06.494>does <00:29:06.602>the <00:29:06.710>action <00:29:06.817>sit <00:29:06.925>and <00:29:07.033>why <00:29:07.140>does <00:29:07.248>it <00:29:07.356>sit <00:29:07.464>over

493
00:29:10.064 --> 00:29:10.344
there?

494
00:29:11.704 --> 00:29:15.564
<00:29:11.704>S <00:29:11.820>for <00:29:11.937>signals. <00:29:12.053>What <00:29:12.170>usage <00:29:12.287>are <00:29:12.404>we <00:29:12.520>logging <00:29:12.637>and <00:29:12.754>which <00:29:12.870>metrics <00:29:12.987>tell <00:29:13.104>us

495
00:29:15.684 --> 00:29:17.904
<00:29:15.684>that <00:29:15.964>it's <00:29:16.244>working? <00:29:16.524>Number <00:29:16.803>five, <00:29:17.084>trust.

496
00:29:18.444 --> 00:29:22.244
<00:29:18.444>How <00:29:18.551>does <00:29:18.659>a <00:29:18.767>user <00:29:18.874>verify <00:29:18.982>an <00:29:19.090>answer <00:29:19.197>and <00:29:19.305>how <00:29:19.413>cheap <00:29:19.520>is <00:29:19.628>it <00:29:19.736>to <00:29:19.844>undo

497
00:29:22.344 --> 00:29:22.524
it?

498
00:29:24.244 --> 00:29:27.414
<00:29:24.244>Now, <00:29:24.399>three <00:29:24.555>things <00:29:24.710>to <00:29:24.866>do <00:29:25.021>this <00:29:25.177>week. <00:29:25.332>Write <00:29:25.488>10 <00:29:25.644>golden

499
00:29:27.524 --> 00:29:30.964
<00:29:27.524>cases. <00:29:27.803>Not <00:29:28.083>the <00:29:28.363>cherry-picked <00:29:28.643>prompts, <00:29:28.924>but

500
00:29:31.004 --> 00:29:34.044
<00:29:31.004>10 <00:29:31.470>messy <00:29:31.937>prompts <00:29:32.404>that

501
00:29:34.944 --> 00:29:36.004
<00:29:34.944>your <00:29:35.262>feature <00:29:35.580>should <00:29:35.898>get.

502
00:29:37.104 --> 00:29:39.224
<00:29:37.104>Add <00:29:37.570>one <00:29:38.037>confidence <00:29:38.504>gate.

503
00:29:40.044 --> 00:29:42.184
<00:29:40.044>Put <00:29:40.394>the <00:29:40.744>riskiest <00:29:41.094>auto <00:29:41.444>action

504
00:29:43.304 --> 00:29:45.204
<00:29:43.304>behind <00:29:43.770>a <00:29:44.237>confidence <00:29:44.704>threshold.

505
00:29:46.414 --> 00:29:49.444
<00:29:46.414>And <00:29:46.647>number <00:29:46.880>three, <00:29:47.114>log <00:29:47.347>one <00:29:47.580>override <00:29:47.814>metric,

506
00:29:50.424 --> 00:29:54.284
<00:29:50.424>which <00:29:50.579>is <00:29:50.735>how <00:29:50.890>often <00:29:51.046>users <00:29:51.201>undo, <00:29:51.357>edit, <00:29:51.512>or <00:29:51.668>ignore <00:29:51.824>the

507
00:29:54.324 --> 00:29:54.724
AI.

508
00:29:55.944 --> 00:29:59.774
<00:29:55.944>And <00:29:56.119>aim <00:29:56.293>for <00:29:56.469>repeated <00:29:56.644>value, <00:29:56.819>not <00:29:56.994>just <00:29:57.168>initial <00:29:57.344>excitement,

509
00:30:00.244 --> 00:30:04.224
<00:30:00.244>because <00:30:00.384>initial <00:30:00.524>excitement <00:30:00.664>is <00:30:00.803>just <00:30:00.944>like <00:30:01.083>a <00:30:01.224>one-time <00:30:01.363>spike <00:30:01.504>and <00:30:01.644>the

510
00:30:04.264 --> 00:30:07.924
<00:30:04.264>real <00:30:04.391>value <00:30:04.518>comes <00:30:04.645>in <00:30:04.773>habit. <00:30:04.900>We <00:30:05.027>are <00:30:05.154>trying <00:30:05.282>to <00:30:05.409>create <00:30:05.536>a <00:30:05.664>compounding

511
00:30:08.044 --> 00:30:11.544
<00:30:08.044>feature <00:30:08.171>that's <00:30:08.298>for <00:30:08.425>the <00:30:08.553>10th <00:30:08.680>time <00:30:08.807>or <00:30:08.934>the <00:30:09.062>20th <00:30:09.189>time <00:30:09.316>someone <00:30:09.444>is

512
00:30:11.664 --> 00:30:15.044
<00:30:11.664>using. <00:30:11.864>So <00:30:12.064>we <00:30:12.264>need <00:30:12.464>to <00:30:12.664>measure <00:30:12.864>repeat <00:30:13.064>usage,

513
00:30:15.184 --> 00:30:18.734
<00:30:15.184>retention. <00:30:15.339>We <00:30:15.495>need <00:30:15.650>to <00:30:15.806>build <00:30:15.961>trust <00:30:16.117>through <00:30:16.272>honesty <00:30:16.428>and <00:30:16.584>recovery.

514
00:30:19.684 --> 00:30:23.384
<00:30:19.684>And <00:30:19.859>we <00:30:20.033>want <00:30:20.209>to <00:30:20.384>aim <00:30:20.559>for <00:30:20.734>that <00:30:20.908>repeated <00:30:21.084>durable

515
00:30:23.444 --> 00:30:24.034
adoption.

516
00:30:26.304 --> 00:30:29.904
<00:30:26.304>If <00:30:26.420>you <00:30:26.537>remember <00:30:26.654>three <00:30:26.770>things, <00:30:26.887>number <00:30:27.004>one, <00:30:27.120>the <00:30:27.237>model <00:30:27.354>is <00:30:27.470>the <00:30:27.587>easy <00:30:27.704>part.

517
00:30:30.004 --> 00:30:33.994
<00:30:30.004>The <00:30:30.178>adoption <00:30:30.353>actually <00:30:30.529>lies <00:30:30.704>in <00:30:30.879>everything <00:30:31.053>around <00:30:31.228>it, <00:30:31.404>just

518
00:30:34.024 --> 00:30:36.664
<00:30:34.024>like <00:30:34.257>we <00:30:34.490>discussed. <00:30:34.724>Trust <00:30:34.957>is <00:30:35.190>the <00:30:35.424>product.

519
00:30:37.584 --> 00:30:40.824
<00:30:37.584>We <00:30:37.739>need <00:30:37.895>to <00:30:38.050>earn <00:30:38.206>that <00:30:38.361>trust <00:30:38.517>so <00:30:38.672>that <00:30:38.828>the <00:30:38.984>users

520
00:30:40.944 --> 00:30:44.124
<00:30:40.944>repeatedly <00:30:41.177>come <00:30:41.410>back <00:30:41.644>and <00:30:41.877>use <00:30:42.110>our <00:30:42.344>product.

521
00:30:44.144 --> 00:30:48.124
<00:30:44.144>And <00:30:44.299>designing <00:30:44.455>it <00:30:44.610>for <00:30:44.766>adoption, <00:30:44.921>for <00:30:45.077>the <00:30:45.232>durable <00:30:45.388>use, <00:30:45.544>for

522
00:30:48.164 --> 00:30:51.484
<00:30:48.164>the <00:30:48.339>retention <00:30:48.514>is <00:30:48.689>what <00:30:48.864>we <00:30:49.039>need <00:30:49.214>to <00:30:49.389>think <00:30:49.564>about.

523
00:30:52.364 --> 00:30:55.484
<00:30:52.364>Thank <00:30:52.564>you <00:30:52.764>so <00:30:52.964>much <00:30:53.164>for <00:30:53.364>attending <00:30:53.564>this <00:30:53.764>presentation.

524
00:30:56.524 --> 00:31:00.364
<00:30:56.524>We <00:30:56.698>need <00:30:56.873>to <00:30:57.049>build <00:30:57.224>AI <00:30:57.399>features <00:30:57.573>that <00:30:57.748>survive <00:30:57.924>the

525
00:31:00.384 --> 00:31:02.524
<00:31:00.384>production, <00:31:00.734>building <00:31:01.084>for <00:31:01.434>the <00:31:01.784>trust.

526
00:31:03.424 --> 00:31:05.524
<00:31:03.424>I'm <00:31:03.657>happy <00:31:03.890>to <00:31:04.124>take <00:31:04.357>this <00:31:04.590>conversation <00:31:04.824>forward.

527
00:31:05.644 --> 00:31:09.064
<00:31:05.644>I've <00:31:05.771>left <00:31:05.898>the <00:31:06.025>link <00:31:06.153>to <00:31:06.280>my <00:31:06.407>website <00:31:06.534>as <00:31:06.662>well <00:31:06.789>as <00:31:06.916>my <00:31:07.044>LinkedIn.

528
00:31:09.444 --> 00:31:12.664
<00:31:09.444>Happy <00:31:09.584>to <00:31:09.724>connect. <00:31:09.864>Thank <00:31:10.004>you <00:31:10.144>so <00:31:10.283>much. <00:31:10.424>Have <00:31:10.563>a <00:31:10.704>great <00:31:10.844>day.