WEBVTT

00:00:02.840 --> 00:00:04.390 align:start position:0%
<00:00:02.840>Good <00:00:03.039>morning. <00:00:03.238>Nice <00:00:03.437>to <00:00:03.637>meet <00:00:03.836>you <00:00:04.035>all. <00:00:04.234>I'm

00:00:04.390 --> 00:00:04.400 align:start position:0%
<00:00:04.390>Good <00:00:04.391>morning. <00:00:04.392>Nice <00:00:04.393>to <00:00:04.395>meet <00:00:04.396>you <00:00:04.397>all. <00:00:04.399>I'm

00:00:04.400 --> 00:00:06.910 align:start position:0%
<00:00:04.400>Good <00:00:04.493>morning. <00:00:04.586>Nice <00:00:04.680>to <00:00:04.773>meet <00:00:04.866>you <00:00:04.960>all. <00:00:05.053>I'm <00:00:05.146>Liran. <00:00:05.240>I'm <00:00:05.333>the <00:00:05.426>VP <00:00:05.520>of <00:00:05.613>AI <00:00:05.706>at <00:00:05.800>Coralogix.

00:00:06.910 --> 00:00:06.920 align:start position:0%
<00:00:06.910>Liran. <00:00:06.911>I'm <00:00:06.912>the <00:00:06.913>VP <00:00:06.915>of <00:00:06.916>AI <00:00:06.917>at <00:00:06.919>Coralogix.

00:00:06.920 --> 00:00:08.510 align:start position:0%
<00:00:06.920>Liran. <00:00:07.013>I'm <00:00:07.106>the <00:00:07.200>VP <00:00:07.293>of <00:00:07.386>AI <00:00:07.479>at <00:00:07.573>Coralogix. <00:00:07.666>And <00:00:07.760>today <00:00:07.853>we <00:00:07.946>are <00:00:08.039>going <00:00:08.133>I'm <00:00:08.226>going <00:00:08.320>to

00:00:08.510 --> 00:00:08.520 align:start position:0%
<00:00:08.510>And <00:00:08.511>today <00:00:08.512>we <00:00:08.513>are <00:00:08.515>going <00:00:08.516>I'm <00:00:08.517>going <00:00:08.519>to

00:00:08.520 --> 00:00:10.350 align:start position:0%
<00:00:08.520>And <00:00:08.627>today <00:00:08.735>we <00:00:08.843>are <00:00:08.950>going <00:00:09.058>I'm <00:00:09.166>going <00:00:09.273>to <00:00:09.381>share <00:00:09.489>with <00:00:09.596>you <00:00:09.704>our <00:00:09.812>experience <00:00:09.920>with

00:00:10.350 --> 00:00:10.360 align:start position:0%
<00:00:10.350>share <00:00:10.351>with <00:00:10.353>you <00:00:10.355>our <00:00:10.357>experience <00:00:10.359>with

00:00:10.360 --> 00:00:12.630 align:start position:0%
<00:00:10.360>share <00:00:10.487>with <00:00:10.614>you <00:00:10.741>our <00:00:10.869>experience <00:00:10.996>with <00:00:11.123>building <00:00:11.250>an <00:00:11.378>agent <00:00:11.505>that <00:00:11.632>is <00:00:11.760>production

00:00:12.630 --> 00:00:12.640 align:start position:0%
<00:00:12.630>building <00:00:12.631>an <00:00:12.633>agent <00:00:12.635>that <00:00:12.637>is <00:00:12.639>production

00:00:12.640 --> 00:00:15.910 align:start position:0%
<00:00:12.640>building <00:00:12.767>an <00:00:12.894>agent <00:00:13.021>that <00:00:13.149>is <00:00:13.276>production <00:00:13.403>facing <00:00:13.530>today <00:00:13.658>serving <00:00:13.785>actually <00:00:13.912>over <00:00:14.040>1,000

00:00:15.910 --> 00:00:15.920 align:start position:0%
<00:00:15.910>facing <00:00:15.911>today <00:00:15.913>serving <00:00:15.915>actually <00:00:15.917>over <00:00:15.919>1,000

00:00:15.920 --> 00:00:19.390 align:start position:0%
<00:00:15.920>facing <00:00:16.059>today <00:00:16.200>serving <00:00:16.340>actually <00:00:16.480>over <00:00:16.620>1,000 <00:00:16.759>customers <00:00:16.900>in <00:00:17.040>production <00:00:17.180>environment. <00:00:17.320>And

00:00:19.390 --> 00:00:19.400 align:start position:0%
<00:00:19.390>customers <00:00:19.392>in <00:00:19.394>production <00:00:19.396>environment. <00:00:19.398>And

00:00:19.400 --> 00:00:21.270 align:start position:0%
<00:00:19.400>customers <00:00:19.599>in <00:00:19.799>production <00:00:20.000>environment. <00:00:20.200>And <00:00:20.400>over <00:00:20.599>this <00:00:20.799>journey,

00:00:21.270 --> 00:00:21.280 align:start position:0%
<00:00:21.270>over <00:00:21.274>this <00:00:21.279>journey,

00:00:21.280 --> 00:00:22.870 align:start position:0%
<00:00:21.280>over <00:00:21.480>this <00:00:21.680>journey, <00:00:21.880>we've <00:00:22.080>had <00:00:22.280>some <00:00:22.480>interesting <00:00:22.680>learning

00:00:22.870 --> 00:00:22.880 align:start position:0%
<00:00:22.870>we've <00:00:22.872>had <00:00:22.874>some <00:00:22.876>interesting <00:00:22.878>learning

00:00:22.880 --> 00:00:24.950 align:start position:0%
<00:00:22.880>we've <00:00:23.007>had <00:00:23.134>some <00:00:23.261>interesting <00:00:23.389>learning <00:00:23.516>experiences <00:00:23.643>I'm <00:00:23.770>going <00:00:23.898>to <00:00:24.025>share <00:00:24.152>with <00:00:24.279>you.

00:00:24.950 --> 00:00:24.960 align:start position:0%
<00:00:24.950>experiences <00:00:24.951>I'm <00:00:24.953>going <00:00:24.954>to <00:00:24.956>share <00:00:24.957>with <00:00:24.959>you.

00:00:24.960 --> 00:00:27.470 align:start position:0%
<00:00:24.960>experiences <00:00:25.067>I'm <00:00:25.175>going <00:00:25.283>to <00:00:25.390>share <00:00:25.498>with <00:00:25.606>you. <00:00:25.713>So, <00:00:25.821>let's <00:00:25.929>start <00:00:26.036>with <00:00:26.144>what <00:00:26.252>is <00:00:26.360>Ollie.

00:00:27.470 --> 00:00:27.480 align:start position:0%
<00:00:27.470>So, <00:00:27.471>let's <00:00:27.473>start <00:00:27.474>with <00:00:27.476>what <00:00:27.477>is <00:00:27.479>Ollie.

00:00:27.480 --> 00:00:30.630 align:start position:0%
<00:00:27.480>So, <00:00:27.587>let's <00:00:27.695>start <00:00:27.803>with <00:00:27.910>what <00:00:28.018>is <00:00:28.126>Ollie. <00:00:28.233>Ollie's <00:00:28.341>our <00:00:28.449>agentic <00:00:28.556>AI <00:00:28.664>which <00:00:28.772>we <00:00:28.880>serve

00:00:30.630 --> 00:00:30.640 align:start position:0%
<00:00:30.630>Ollie's <00:00:30.631>our <00:00:30.633>agentic <00:00:30.634>AI <00:00:30.636>which <00:00:30.637>we <00:00:30.639>serve

00:00:30.640 --> 00:00:32.510 align:start position:0%
<00:00:30.640>Ollie's <00:00:30.756>our <00:00:30.873>agentic <00:00:30.990>AI <00:00:31.106>which <00:00:31.223>we <00:00:31.340>serve <00:00:31.456>for <00:00:31.573>our <00:00:31.690>customers. <00:00:31.806>Maybe <00:00:31.923>a <00:00:32.040>step

00:00:32.510 --> 00:00:32.520 align:start position:0%
<00:00:32.509>for <00:00:32.511>our <00:00:32.513>customers. <00:00:32.515>Maybe <00:00:32.517>a <00:00:32.519>step

00:00:32.520 --> 00:00:34.950 align:start position:0%
<00:00:32.520>for <00:00:32.660>our <00:00:32.800>customers. <00:00:32.940>Maybe <00:00:33.080>a <00:00:33.220>step <00:00:33.360>backward. <00:00:33.500>Coralogix <00:00:33.640>is <00:00:33.780>an <00:00:33.920>observability

00:00:34.950 --> 00:00:34.960 align:start position:0%
<00:00:34.950>backward. <00:00:34.952>Coralogix <00:00:34.954>is <00:00:34.956>an <00:00:34.959>observability

00:00:34.960 --> 00:00:37.710 align:start position:0%
<00:00:34.960>backward. <00:00:35.100>Coralogix <00:00:35.240>is <00:00:35.380>an <00:00:35.520>observability <00:00:35.660>platform. <00:00:35.800>So, <00:00:35.940>you <00:00:36.080>send <00:00:36.220>logs, <00:00:36.360>metrics,

00:00:37.710 --> 00:00:37.720 align:start position:0%
<00:00:37.710>platform. <00:00:37.711>So, <00:00:37.713>you <00:00:37.715>send <00:00:37.717>logs, <00:00:37.719>metrics,

00:00:37.720 --> 00:00:39.110 align:start position:0%
<00:00:37.720>platform. <00:00:37.928>So, <00:00:38.137>you <00:00:38.345>send <00:00:38.554>logs, <00:00:38.762>metrics, <00:00:38.971>traces

00:00:39.110 --> 00:00:39.120 align:start position:0%
traces

00:00:39.120 --> 00:00:41.750 align:start position:0%
<00:00:39.120>traces <00:00:39.353>every <00:00:39.586>production <00:00:39.820>telemetry <00:00:40.053>that <00:00:40.286>you <00:00:40.519>have

00:00:41.750 --> 00:00:41.760 align:start position:0%
<00:00:41.750>every <00:00:41.751>production <00:00:41.753>telemetry <00:00:41.755>that <00:00:41.757>you <00:00:41.759>have

00:00:41.760 --> 00:00:44.670 align:start position:0%
<00:00:41.760>every <00:00:41.876>production <00:00:41.993>telemetry <00:00:42.110>that <00:00:42.226>you <00:00:42.343>have <00:00:42.460>to <00:00:42.576>Coralogix. <00:00:42.693>And <00:00:42.809>then <00:00:42.926>you <00:00:43.043>can <00:00:43.160>build

00:00:44.670 --> 00:00:44.680 align:start position:0%
<00:00:44.670>to <00:00:44.671>Coralogix. <00:00:44.673>And <00:00:44.674>then <00:00:44.676>you <00:00:44.677>can <00:00:44.679>build

00:00:44.680 --> 00:00:46.910 align:start position:0%
<00:00:44.680>to <00:00:44.807>Coralogix. <00:00:44.934>And <00:00:45.061>then <00:00:45.189>you <00:00:45.316>can <00:00:45.443>build <00:00:45.570>dashboards, <00:00:45.698>monitor, <00:00:45.825>track <00:00:45.952>them, <00:00:46.080>get

00:00:46.910 --> 00:00:46.920 align:start position:0%
<00:00:46.910>dashboards, <00:00:46.912>monitor, <00:00:46.914>track <00:00:46.916>them, <00:00:46.919>get

00:00:46.920 --> 00:00:49.310 align:start position:0%
<00:00:46.920>dashboards, <00:00:47.047>monitor, <00:00:47.174>track <00:00:47.301>them, <00:00:47.429>get <00:00:47.556>alerts, <00:00:47.683>identify <00:00:47.810>what <00:00:47.938>is <00:00:48.065>the <00:00:48.192>root <00:00:48.320>cause

00:00:49.310 --> 00:00:49.320 align:start position:0%
<00:00:49.310>alerts, <00:00:49.311>identify <00:00:49.313>what <00:00:49.314>is <00:00:49.316>the <00:00:49.317>root <00:00:49.319>cause

00:00:49.320 --> 00:00:51.230 align:start position:0%
<00:00:49.320>alerts, <00:00:49.436>identify <00:00:49.553>what <00:00:49.670>is <00:00:49.786>the <00:00:49.903>root <00:00:50.020>cause <00:00:50.136>and <00:00:50.253>so <00:00:50.370>on <00:00:50.486>and <00:00:50.603>so <00:00:50.720>forth.

00:00:51.230 --> 00:00:51.240 align:start position:0%
<00:00:51.230>and <00:00:51.231>so <00:00:51.233>on <00:00:51.235>and <00:00:51.237>so <00:00:51.239>forth.

00:00:51.240 --> 00:00:53.470 align:start position:0%
<00:00:51.240>and <00:00:51.395>so <00:00:51.551>on <00:00:51.706>and <00:00:51.862>so <00:00:52.017>forth. <00:00:52.173>Having <00:00:52.328>all <00:00:52.484>these <00:00:52.640>data

00:00:53.470 --> 00:00:53.480 align:start position:0%
<00:00:53.470>Having <00:00:53.473>all <00:00:53.476>these <00:00:53.479>data

00:00:53.480 --> 00:00:56.470 align:start position:0%
<00:00:53.480>Having <00:00:53.635>all <00:00:53.791>these <00:00:53.946>data <00:00:54.102>is <00:00:54.257>huge <00:00:54.413>has <00:00:54.568>the <00:00:54.724>huge <00:00:54.879>potential

00:00:56.470 --> 00:00:56.480 align:start position:0%
<00:00:56.470>is <00:00:56.471>huge <00:00:56.473>has <00:00:56.475>the <00:00:56.477>huge <00:00:56.479>potential

00:00:56.480 --> 00:00:58.630 align:start position:0%
<00:00:56.480>is <00:00:56.587>huge <00:00:56.695>has <00:00:56.803>the <00:00:56.910>huge <00:00:57.018>potential <00:00:57.126>to <00:00:57.233>actually <00:00:57.341>get <00:00:57.449>a <00:00:57.556>lot <00:00:57.664>of <00:00:57.772>insights <00:00:57.879>with

00:00:58.630 --> 00:00:58.640 align:start position:0%
<00:00:58.630>to <00:00:58.631>actually <00:00:58.632>get <00:00:58.633>a <00:00:58.635>lot <00:00:58.636>of <00:00:58.637>insights <00:00:58.639>with

00:00:58.640 --> 00:00:59.710 align:start position:0%
<00:00:58.640>to <00:00:58.760>actually <00:00:58.880>get <00:00:59.001>a <00:00:59.121>lot <00:00:59.241>of <00:00:59.362>insights <00:00:59.482>with <00:00:59.603>AI.

00:00:59.710 --> 00:00:59.720 align:start position:0%
AI.

00:00:59.720 --> 00:01:02.190 align:start position:0%
<00:00:59.720>AI. <00:01:01.120>So,

00:01:02.200 --> 00:01:04.189 align:start position:0%
<00:01:02.200>when <00:01:02.375>we <00:01:02.550>started, <00:01:02.725>so <00:01:02.900>this <00:01:03.075>is <00:01:03.250>kind <00:01:03.425>of <00:01:03.600>how

00:01:04.189 --> 00:01:04.199 align:start position:0%
<00:01:04.188>when <00:01:04.190>we <00:01:04.191>started, <00:01:04.192>so <00:01:04.193>this <00:01:04.194>is <00:01:04.195>kind <00:01:04.196>of <00:01:04.197>how

00:01:04.199 --> 00:01:05.750 align:start position:0%
<00:01:04.199>when <00:01:04.286>we <00:01:04.373>started, <00:01:04.460>so <00:01:04.547>this <00:01:04.635>is <00:01:04.722>kind <00:01:04.809>of <00:01:04.896>how <00:01:04.984>we <00:01:05.071>started <00:01:05.158>Ollie. <00:01:05.245>You <00:01:05.333>see <00:01:05.420>Ollie <00:01:05.507>on <00:01:05.594>the

00:01:05.750 --> 00:01:05.760 align:start position:0%
<00:01:05.750>we <00:01:05.751>started <00:01:05.752>Ollie. <00:01:05.753>You <00:01:05.755>see <00:01:05.756>Ollie <00:01:05.757>on <00:01:05.759>the

00:01:05.760 --> 00:01:07.230 align:start position:0%
<00:01:05.760>we <00:01:05.854>started <00:01:05.949>Ollie. <00:01:06.043>You <00:01:06.138>see <00:01:06.232>Ollie <00:01:06.327>on <00:01:06.421>the <00:01:06.516>screen. <00:01:06.610>The <00:01:06.705>when <00:01:06.799>what <00:01:06.894>Ollie <00:01:06.988>is <00:01:07.083>doing,

00:01:07.230 --> 00:01:07.240 align:start position:0%
<00:01:07.230>screen. <00:01:07.231>The <00:01:07.233>when <00:01:07.234>what <00:01:07.236>Ollie <00:01:07.237>is <00:01:07.238>doing,

00:01:07.240 --> 00:01:09.670 align:start position:0%
<00:01:07.240>screen. <00:01:07.347>The <00:01:07.455>when <00:01:07.563>what <00:01:07.670>Ollie <00:01:07.778>is <00:01:07.886>doing, <00:01:07.993>Ollie's <00:01:08.101>an <00:01:08.209>agentic <00:01:08.316>system. <00:01:08.424>It's <00:01:08.532>not <00:01:08.640>just

00:01:09.670 --> 00:01:09.680 align:start position:0%
<00:01:09.670>Ollie's <00:01:09.671>an <00:01:09.673>agentic <00:01:09.674>system. <00:01:09.676>It's <00:01:09.677>not <00:01:09.679>just

00:01:09.680 --> 00:01:11.550 align:start position:0%
<00:01:09.680>Ollie's <00:01:09.787>an <00:01:09.895>agentic <00:01:10.003>system. <00:01:10.110>It's <00:01:10.218>not <00:01:10.326>just <00:01:10.433>one <00:01:10.541>agent <00:01:10.649>behind <00:01:10.756>the <00:01:10.864>scenes. <00:01:10.972>There <00:01:11.080>are

00:01:11.550 --> 00:01:11.560 align:start position:0%
<00:01:11.550>one <00:01:11.551>agent <00:01:11.553>behind <00:01:11.554>the <00:01:11.556>scenes. <00:01:11.557>There <00:01:11.559>are

00:01:11.560 --> 00:01:13.510 align:start position:0%
<00:01:11.560>one <00:01:11.700>agent <00:01:11.840>behind <00:01:11.980>the <00:01:12.120>scenes. <00:01:12.260>There <00:01:12.400>are <00:01:12.540>multiple <00:01:12.680>sub <00:01:12.820>agents <00:01:12.960>collaborating

00:01:13.510 --> 00:01:13.520 align:start position:0%
<00:01:13.510>multiple <00:01:13.513>sub <00:01:13.516>agents <00:01:13.518>collaborating

00:01:13.520 --> 00:01:15.670 align:start position:0%
<00:01:13.520>multiple <00:01:13.660>sub <00:01:13.800>agents <00:01:13.940>collaborating <00:01:14.080>together <00:01:14.220>on <00:01:14.360>top <00:01:14.500>of <00:01:14.640>the <00:01:14.780>knowledge <00:01:14.920>that

00:01:15.670 --> 00:01:15.680 align:start position:0%
<00:01:15.670>together <00:01:15.671>on <00:01:15.673>top <00:01:15.674>of <00:01:15.676>the <00:01:15.677>knowledge <00:01:15.679>that

00:01:15.680 --> 00:01:16.990 align:start position:0%
<00:01:15.680>together <00:01:15.827>on <00:01:15.974>top <00:01:16.122>of <00:01:16.269>the <00:01:16.416>knowledge <00:01:16.564>that <00:01:16.711>we've <00:01:16.859>built.

00:01:16.990 --> 00:01:17.000 align:start position:0%
<00:01:16.990>we've <00:01:16.999>built.

00:01:17.000 --> 00:01:17.990 align:start position:0%
<00:01:17.000>we've <00:01:17.445>built. <00:01:17.890>Um

00:01:17.990 --> 00:01:18.000 align:start position:0%
Um

00:01:18.000 --> 00:01:19.870 align:start position:0%
<00:01:18.000>Um <00:01:18.175>and <00:01:18.350>what <00:01:18.525>you <00:01:18.700>see <00:01:18.875>here <00:01:19.050>is <00:01:19.225>actually <00:01:19.400>an

00:01:19.870 --> 00:01:19.880 align:start position:0%
<00:01:19.870>and <00:01:19.871>what <00:01:19.872>you <00:01:19.873>see <00:01:19.875>here <00:01:19.876>is <00:01:19.877>actually <00:01:19.878>an

00:01:19.880 --> 00:01:22.510 align:start position:0%
<00:01:19.880>and <00:01:19.973>what <00:01:20.066>you <00:01:20.160>see <00:01:20.253>here <00:01:20.346>is <00:01:20.440>actually <00:01:20.533>an <00:01:20.626>example <00:01:20.720>of <00:01:20.813>a <00:01:20.906>user <00:01:21.000>coming <00:01:21.093>in <00:01:21.186>and <00:01:21.280>instead

00:01:22.510 --> 00:01:22.520 align:start position:0%
<00:01:22.510>example <00:01:22.511>of <00:01:22.512>a <00:01:22.513>user <00:01:22.515>coming <00:01:22.516>in <00:01:22.517>and <00:01:22.518>instead

00:01:22.520 --> 00:01:24.630 align:start position:0%
<00:01:22.520>example <00:01:22.636>of <00:01:22.753>a <00:01:22.869>user <00:01:22.986>coming <00:01:23.103>in <00:01:23.220>and <00:01:23.336>instead <00:01:23.453>of <00:01:23.570>manually <00:01:23.686>investigating <00:01:23.803>a <00:01:23.920>production

00:01:24.630 --> 00:01:24.640 align:start position:0%
<00:01:24.630>of <00:01:24.632>manually <00:01:24.634>investigating <00:01:24.636>a <00:01:24.639>production

00:01:24.640 --> 00:01:27.110 align:start position:0%
<00:01:24.640>of <00:01:24.815>manually <00:01:24.990>investigating <00:01:25.165>a <00:01:25.340>production <00:01:25.515>issue, <00:01:25.690>just <00:01:25.865>sharing, <00:01:26.040>"Hey,

00:01:27.110 --> 00:01:27.120 align:start position:0%
<00:01:27.110>issue, <00:01:27.113>just <00:01:27.116>sharing, <00:01:27.119>"Hey,

00:01:27.120 --> 00:01:29.990 align:start position:0%
<00:01:27.120>issue, <00:01:27.275>just <00:01:27.431>sharing, <00:01:27.586>"Hey, <00:01:27.742>users <00:01:27.897>complain <00:01:28.053>on <00:01:28.208>slowness <00:01:28.364>in <00:01:28.520>Europe.

00:01:29.990 --> 00:01:30.000 align:start position:0%
<00:01:29.990>users <00:01:29.991>complain <00:01:29.993>on <00:01:29.995>slowness <00:01:29.997>in <00:01:29.999>Europe.

00:01:30.000 --> 00:01:31.430 align:start position:0%
<00:01:30.000>users <00:01:30.107>complain <00:01:30.214>on <00:01:30.321>slowness <00:01:30.429>in <00:01:30.536>Europe. <00:01:30.643>Can <00:01:30.750>you <00:01:30.858>help <00:01:30.965>me <00:01:31.072>out <00:01:31.179>understand <00:01:31.287>what's

00:01:31.430 --> 00:01:31.440 align:start position:0%
<00:01:31.430>Can <00:01:31.431>you <00:01:31.433>help <00:01:31.434>me <00:01:31.436>out <00:01:31.437>understand <00:01:31.439>what's

00:01:31.440 --> 00:01:32.550 align:start position:0%
<00:01:31.440>Can <00:01:31.564>you <00:01:31.689>help <00:01:31.814>me <00:01:31.939>out <00:01:32.064>understand <00:01:32.189>what's <00:01:32.314>going <00:01:32.439>on?"

00:01:32.550 --> 00:01:32.560 align:start position:0%
<00:01:32.550>going <00:01:32.559>on?"

00:01:32.560 --> 00:01:35.310 align:start position:0%
<00:01:32.560>going <00:01:32.715>on?" <00:01:32.871>And <00:01:33.026>Ollie <00:01:33.182>would <00:01:33.337>go <00:01:33.493>behind <00:01:33.648>the <00:01:33.804>scenes <00:01:33.960>and

00:01:35.310 --> 00:01:35.320 align:start position:0%
<00:01:35.310>And <00:01:35.311>Ollie <00:01:35.312>would <00:01:35.313>go <00:01:35.315>behind <00:01:35.316>the <00:01:35.317>scenes <00:01:35.318>and

00:01:35.320 --> 00:01:37.270 align:start position:0%
<00:01:35.320>And <00:01:35.436>Ollie <00:01:35.553>would <00:01:35.669>go <00:01:35.786>behind <00:01:35.903>the <00:01:36.020>scenes <00:01:36.136>and <00:01:36.253>query <00:01:36.369>logs, <00:01:36.486>query <00:01:36.603>metrics, <00:01:36.720>create

00:01:37.270 --> 00:01:37.280 align:start position:0%
<00:01:37.270>query <00:01:37.272>logs, <00:01:37.274>query <00:01:37.276>metrics, <00:01:37.279>create

00:01:37.280 --> 00:01:39.950 align:start position:0%
<00:01:37.280>query <00:01:37.455>logs, <00:01:37.630>query <00:01:37.805>metrics, <00:01:37.980>create <00:01:38.155>correlation <00:01:38.330>with <00:01:38.505>traces, <00:01:38.680>identify

00:01:39.950 --> 00:01:39.960 align:start position:0%
<00:01:39.950>correlation <00:01:39.953>with <00:01:39.956>traces, <00:01:39.959>identify

00:01:39.960 --> 00:01:42.310 align:start position:0%
<00:01:39.960>correlation <00:01:40.087>with <00:01:40.214>traces, <00:01:40.341>identify <00:01:40.469>hopefully <00:01:40.596>what <00:01:40.723>is <00:01:40.850>the <00:01:40.978>root <00:01:41.105>cause <00:01:41.232>or <00:01:41.360>for

00:01:42.310 --> 00:01:42.320 align:start position:0%
<00:01:42.310>hopefully <00:01:42.311>what <00:01:42.312>is <00:01:42.313>the <00:01:42.315>root <00:01:42.316>cause <00:01:42.317>or <00:01:42.318>for

00:01:42.320 --> 00:01:44.590 align:start position:0%
<00:01:42.320>hopefully <00:01:42.413>what <00:01:42.506>is <00:01:42.600>the <00:01:42.693>root <00:01:42.786>cause <00:01:42.880>or <00:01:42.973>for <00:01:43.066>the <00:01:43.160>very <00:01:43.253>least <00:01:43.346>help <00:01:43.440>you <00:01:43.533>and <00:01:43.626>shortcut <00:01:43.720>the

00:01:44.590 --> 00:01:44.600 align:start position:0%
<00:01:44.590>the <00:01:44.591>very <00:01:44.592>least <00:01:44.593>help <00:01:44.595>you <00:01:44.596>and <00:01:44.597>shortcut <00:01:44.598>the

00:01:44.600 --> 00:01:47.430 align:start position:0%
<00:01:44.600>the <00:01:44.707>very <00:01:44.815>least <00:01:44.923>help <00:01:45.030>you <00:01:45.138>and <00:01:45.246>shortcut <00:01:45.353>the <00:01:45.461>way <00:01:45.569>to <00:01:45.676>finding <00:01:45.784>the <00:01:45.892>root <00:01:46.000>cause.

00:01:47.430 --> 00:01:47.440 align:start position:0%
<00:01:47.430>way <00:01:47.431>to <00:01:47.433>finding <00:01:47.435>the <00:01:47.437>root <00:01:47.439>cause.

00:01:47.440 --> 00:01:50.870 align:start position:0%
<00:01:47.440>way <00:01:47.673>to <00:01:47.906>finding <00:01:48.140>the <00:01:48.373>root <00:01:48.606>cause. <00:01:48.840>So,

00:01:50.880 --> 00:01:53.550 align:start position:0%
<00:01:50.880>how <00:01:51.113>how <00:01:51.346>we <00:01:51.580>actually <00:01:51.813>started <00:01:52.046>with <00:01:52.280>Ollie.

00:01:53.550 --> 00:01:53.560 align:start position:0%
<00:01:53.550>how <00:01:53.551>how <00:01:53.553>we <00:01:53.554>actually <00:01:53.556>started <00:01:53.557>with <00:01:53.559>Ollie.

00:01:53.560 --> 00:01:56.870 align:start position:0%
<00:01:53.560>how <00:01:53.700>how <00:01:53.840>we <00:01:53.980>actually <00:01:54.120>started <00:01:54.260>with <00:01:54.400>Ollie. <00:01:54.540>This <00:01:54.680>was <00:01:54.820>January <00:01:54.960>2025.

00:01:56.870 --> 00:01:56.880 align:start position:0%
<00:01:56.870>This <00:01:56.873>was <00:01:56.876>January <00:01:56.878>2025.

00:01:56.880 --> 00:01:59.150 align:start position:0%
<00:01:56.880>This <00:01:57.007>was <00:01:57.134>January <00:01:57.261>2025. <00:01:57.389>Beforehand <00:01:57.516>I <00:01:57.643>was <00:01:57.770>the <00:01:57.898>CEO <00:01:58.025>of <00:01:58.152>Aporia <00:01:58.280>where

00:01:59.150 --> 00:01:59.160 align:start position:0%
<00:01:59.150>Beforehand <00:01:59.151>I <00:01:59.152>was <00:01:59.153>the <00:01:59.155>CEO <00:01:59.156>of <00:01:59.157>Aporia <00:01:59.158>where

00:01:59.160 --> 00:02:01.190 align:start position:0%
<00:01:59.160>Beforehand <00:01:59.276>I <00:01:59.393>was <00:01:59.509>the <00:01:59.626>CEO <00:01:59.743>of <00:01:59.860>Aporia <00:01:59.976>where <00:02:00.093>we've <00:02:00.210>done <00:02:00.326>observe <00:02:00.443>AI <00:02:00.560>observability

00:02:01.190 --> 00:02:01.200 align:start position:0%
<00:02:01.190>we've <00:02:01.192>done <00:02:01.194>observe <00:02:01.196>AI <00:02:01.199>observability

00:02:01.200 --> 00:02:02.550 align:start position:0%
<00:02:01.200>we've <00:02:01.321>done <00:02:01.443>observe <00:02:01.564>AI <00:02:01.686>observability <00:02:01.807>platform <00:02:01.929>and <00:02:02.050>we <00:02:02.172>got <00:02:02.293>acquired <00:02:02.414>by

00:02:02.550 --> 00:02:02.560 align:start position:0%
<00:02:02.550>platform <00:02:02.551>and <00:02:02.553>we <00:02:02.555>got <00:02:02.557>acquired <00:02:02.559>by

00:02:02.560 --> 00:02:05.590 align:start position:0%
<00:02:02.560>platform <00:02:02.687>and <00:02:02.814>we <00:02:02.941>got <00:02:03.069>acquired <00:02:03.196>by <00:02:03.323>Coralogix. <00:02:03.450>So, <00:02:03.578>we <00:02:03.705>just <00:02:03.832>got <00:02:03.960>acquired.

00:02:05.590 --> 00:02:05.600 align:start position:0%
<00:02:05.590>Coralogix. <00:02:05.591>So, <00:02:05.593>we <00:02:05.595>just <00:02:05.597>got <00:02:05.598>acquired.

00:02:05.600 --> 00:02:07.430 align:start position:0%
<00:02:05.600>Coralogix. <00:02:05.707>So, <00:02:05.815>we <00:02:05.923>just <00:02:06.030>got <00:02:06.138>acquired. <00:02:06.246>Our <00:02:06.353>first <00:02:06.461>2 <00:02:06.569>weeks <00:02:06.676>at <00:02:06.784>Coralogix <00:02:06.892>at <00:02:07.000>the

00:02:07.430 --> 00:02:07.440 align:start position:0%
<00:02:07.430>Our <00:02:07.431>first <00:02:07.432>2 <00:02:07.433>weeks <00:02:07.435>at <00:02:07.436>Coralogix <00:02:07.437>at <00:02:07.439>the

00:02:07.440 --> 00:02:08.669 align:start position:0%
<00:02:07.440>Our <00:02:07.578>first <00:02:07.716>2 <00:02:07.854>weeks <00:02:07.993>at <00:02:08.131>Coralogix <00:02:08.269>at <00:02:08.407>the <00:02:08.546>company,

00:02:08.669 --> 00:02:08.679 align:start position:0%
company,

00:02:08.679 --> 00:02:10.790 align:start position:0%
<00:02:08.679>company, <00:02:08.854>um <00:02:09.029>and <00:02:09.204>I <00:02:09.378>remember <00:02:09.554>we <00:02:09.729>have <00:02:09.904>this <00:02:10.079>brilliant

00:02:10.790 --> 00:02:10.800 align:start position:0%
<00:02:10.789>um <00:02:10.791>and <00:02:10.792>I <00:02:10.793>remember <00:02:10.795>we <00:02:10.796>have <00:02:10.797>this <00:02:10.799>brilliant

00:02:10.800 --> 00:02:14.030 align:start position:0%
<00:02:10.800>um <00:02:10.907>and <00:02:11.015>I <00:02:11.123>remember <00:02:11.230>we <00:02:11.338>have <00:02:11.446>this <00:02:11.553>brilliant <00:02:11.661>vision <00:02:11.769>of <00:02:11.876>what <00:02:11.984>an <00:02:12.092>observability <00:02:12.200>agent

00:02:14.030 --> 00:02:14.040 align:start position:0%
<00:02:14.030>vision <00:02:14.031>of <00:02:14.033>what <00:02:14.035>an <00:02:14.037>observability <00:02:14.039>agent

00:02:14.040 --> 00:02:15.070 align:start position:0%
<00:02:14.040>vision <00:02:14.172>of <00:02:14.304>what <00:02:14.437>an <00:02:14.569>observability <00:02:14.702>agent <00:02:14.834>could <00:02:14.966>do.

00:02:15.070 --> 00:02:15.080 align:start position:0%
<00:02:15.070>could <00:02:15.079>do.

00:02:15.080 --> 00:02:17.230 align:start position:0%
<00:02:15.080>could <00:02:15.235>do. <00:02:15.391>Now, <00:02:15.546>I <00:02:15.702>know <00:02:15.857>today <00:02:16.013>it <00:02:16.168>may <00:02:16.324>sound <00:02:16.480>obvious,

00:02:17.230 --> 00:02:17.240 align:start position:0%
<00:02:17.230>Now, <00:02:17.231>I <00:02:17.232>know <00:02:17.233>today <00:02:17.235>it <00:02:17.236>may <00:02:17.237>sound <00:02:17.239>obvious,

00:02:17.240 --> 00:02:18.910 align:start position:0%
<00:02:17.240>Now, <00:02:17.340>I <00:02:17.440>know <00:02:17.540>today <00:02:17.640>it <00:02:17.740>may <00:02:17.840>sound <00:02:17.940>obvious, <00:02:18.040>yeah, <00:02:18.140>like <00:02:18.240>it's <00:02:18.340>clear <00:02:18.440>that <00:02:18.540>what <00:02:18.640>agents

00:02:18.910 --> 00:02:18.920 align:start position:0%
<00:02:18.910>yeah, <00:02:18.911>like <00:02:18.913>it's <00:02:18.914>clear <00:02:18.916>that <00:02:18.917>what <00:02:18.919>agents

00:02:18.920 --> 00:02:22.270 align:start position:0%
<00:02:18.920>yeah, <00:02:19.007>like <00:02:19.095>it's <00:02:19.182>clear <00:02:19.270>that <00:02:19.357>what <00:02:19.445>agents <00:02:19.532>can <00:02:19.620>do. <00:02:19.707>But <00:02:19.795>yes, <00:02:19.882>1 <00:02:19.970>and <00:02:20.057>1/2 <00:02:20.145>years <00:02:20.232>ago, <00:02:20.320>it

00:02:22.270 --> 00:02:22.280 align:start position:0%
<00:02:22.270>can <00:02:22.271>do. <00:02:22.272>But <00:02:22.273>yes, <00:02:22.274>1 <00:02:22.275>and <00:02:22.276>1/2 <00:02:22.277>years <00:02:22.278>ago, <00:02:22.279>it

00:02:22.280 --> 00:02:23.670 align:start position:0%
<00:02:22.280>can <00:02:22.363>do. <00:02:22.446>But <00:02:22.530>yes, <00:02:22.613>1 <00:02:22.697>and <00:02:22.780>1/2 <00:02:22.863>years <00:02:22.947>ago, <00:02:23.030>it <00:02:23.114>sounded <00:02:23.197>like, <00:02:23.280>"Wow, <00:02:23.364>that <00:02:23.447>could <00:02:23.531>be

00:02:23.670 --> 00:02:23.680 align:start position:0%
<00:02:23.670>sounded <00:02:23.671>like, <00:02:23.673>"Wow, <00:02:23.675>that <00:02:23.677>could <00:02:23.679>be

00:02:23.680 --> 00:02:25.630 align:start position:0%
<00:02:23.680>sounded <00:02:23.796>like, <00:02:23.913>"Wow, <00:02:24.030>that <00:02:24.146>could <00:02:24.263>be <00:02:24.380>amazing <00:02:24.496>if <00:02:24.613>we <00:02:24.730>would <00:02:24.846>have <00:02:24.963>agent <00:02:25.080>that

00:02:25.630 --> 00:02:25.640 align:start position:0%
<00:02:25.630>amazing <00:02:25.631>if <00:02:25.632>we <00:02:25.634>would <00:02:25.636>have <00:02:25.637>agent <00:02:25.638>that

00:02:25.640 --> 00:02:27.870 align:start position:0%
<00:02:25.640>amazing <00:02:25.767>if <00:02:25.894>we <00:02:26.021>would <00:02:26.149>have <00:02:26.276>agent <00:02:26.403>that <00:02:26.530>would <00:02:26.658>do <00:02:26.785>that, <00:02:26.912>right?" <00:02:27.040>Um

00:02:27.870 --> 00:02:27.880 align:start position:0%
<00:02:27.870>would <00:02:27.872>do <00:02:27.874>that, <00:02:27.876>right?" <00:02:27.879>Um

00:02:27.880 --> 00:02:28.990 align:start position:0%
<00:02:27.880>would <00:02:28.079>do <00:02:28.279>that, <00:02:28.479>right?" <00:02:28.679>Um <00:02:28.879>So,

00:02:28.990 --> 00:02:29.000 align:start position:0%
So,

00:02:29.000 --> 00:02:31.590 align:start position:0%
<00:02:29.000>So, <00:02:29.175>we <00:02:29.350>built <00:02:29.525>a <00:02:29.700>quite <00:02:29.875>an <00:02:30.050>impressive <00:02:30.225>demo. <00:02:30.400>Um

00:02:31.590 --> 00:02:31.600 align:start position:0%
<00:02:31.590>we <00:02:31.591>built <00:02:31.592>a <00:02:31.593>quite <00:02:31.595>an <00:02:31.596>impressive <00:02:31.597>demo. <00:02:31.599>Um

00:02:31.600 --> 00:02:33.630 align:start position:0%
<00:02:31.600>we <00:02:31.707>built <00:02:31.815>a <00:02:31.923>quite <00:02:32.030>an <00:02:32.138>impressive <00:02:32.246>demo. <00:02:32.353>Um <00:02:32.461>honestly, <00:02:32.569>it <00:02:32.676>took <00:02:32.784>us <00:02:32.892>2 <00:02:33.000>weeks

00:02:33.630 --> 00:02:33.640 align:start position:0%
<00:02:33.630>honestly, <00:02:33.631>it <00:02:33.633>took <00:02:33.635>us <00:02:33.637>2 <00:02:33.638>weeks

00:02:33.640 --> 00:02:36.070 align:start position:0%
<00:02:33.640>honestly, <00:02:33.747>it <00:02:33.855>took <00:02:33.963>us <00:02:34.070>2 <00:02:34.178>weeks <00:02:34.286>of <00:02:34.393>Ollie <00:02:34.501>which <00:02:34.609>was <00:02:34.716>capable <00:02:34.824>that <00:02:34.932>in <00:02:35.040>the

00:02:36.070 --> 00:02:36.080 align:start position:0%
<00:02:36.070>of <00:02:36.071>Ollie <00:02:36.072>which <00:02:36.073>was <00:02:36.075>capable <00:02:36.076>that <00:02:36.077>in <00:02:36.078>the

00:02:36.080 --> 00:02:37.910 align:start position:0%
<00:02:36.079>of <00:02:36.179>Ollie <00:02:36.279>which <00:02:36.380>was <00:02:36.480>capable <00:02:36.579>that <00:02:36.679>in <00:02:36.779>the <00:02:36.880>demo <00:02:36.980>you <00:02:37.079>could <00:02:37.179>have <00:02:37.279>seen <00:02:37.380>um <00:02:37.480>logs,

00:02:37.910 --> 00:02:37.920 align:start position:0%
<00:02:37.910>demo <00:02:37.911>you <00:02:37.913>could <00:02:37.914>have <00:02:37.916>seen <00:02:37.917>um <00:02:37.919>logs,

00:02:37.920 --> 00:02:40.190 align:start position:0%
<00:02:37.920>demo <00:02:38.075>you <00:02:38.231>could <00:02:38.386>have <00:02:38.542>seen <00:02:38.697>um <00:02:38.853>logs, <00:02:39.008>metrics, <00:02:39.164>traces, <00:02:39.320>investigation,

00:02:40.190 --> 00:02:40.200 align:start position:0%
<00:02:40.190>metrics, <00:02:40.194>traces, <00:02:40.198>investigation,

00:02:40.200 --> 00:02:42.750 align:start position:0%
<00:02:40.200>metrics, <00:02:40.355>traces, <00:02:40.511>investigation, <00:02:40.666>integration <00:02:40.822>with <00:02:40.977>code <00:02:41.133>base <00:02:41.288>in <00:02:41.444>in <00:02:41.600>GitHub

00:02:42.750 --> 00:02:42.760 align:start position:0%
<00:02:42.750>integration <00:02:42.751>with <00:02:42.753>code <00:02:42.754>base <00:02:42.756>in <00:02:42.757>in <00:02:42.759>GitHub

00:02:42.760 --> 00:02:43.830 align:start position:0%
<00:02:42.760>integration <00:02:42.897>with <00:02:43.035>code <00:02:43.172>base <00:02:43.310>in <00:02:43.447>in <00:02:43.585>GitHub <00:02:43.722>repos,

00:02:43.830 --> 00:02:43.840 align:start position:0%
repos,

00:02:43.840 --> 00:02:44.590 align:start position:0%
<00:02:43.840>repos, <00:02:44.515>um

00:02:44.590 --> 00:02:44.600 align:start position:0%
um

00:02:44.600 --> 00:02:46.310 align:start position:0%
<00:02:44.600>um <00:02:44.799>investigation <00:02:45.000>of <00:02:45.200>all <00:02:45.400>of <00:02:45.600>that <00:02:45.799>and <00:02:46.000>getting

00:02:46.310 --> 00:02:46.320 align:start position:0%
<00:02:46.310>investigation <00:02:46.311>of <00:02:46.313>all <00:02:46.314>of <00:02:46.316>that <00:02:46.317>and <00:02:46.319>getting

00:02:46.320 --> 00:02:47.949 align:start position:0%
<00:02:46.320>investigation <00:02:46.459>of <00:02:46.600>all <00:02:46.739>of <00:02:46.880>that <00:02:47.019>and <00:02:47.160>getting <00:02:47.299>to <00:02:47.440>the <00:02:47.579>root <00:02:47.720>cause.

00:02:47.949 --> 00:02:47.959 align:start position:0%
<00:02:47.949>to <00:02:47.952>the <00:02:47.955>root <00:02:47.958>cause.

00:02:47.959 --> 00:02:50.750 align:start position:0%
<00:02:47.959>to <00:02:48.099>the <00:02:48.239>root <00:02:48.379>cause. <00:02:48.519>And <00:02:48.659>still, <00:02:48.799>even <00:02:48.939>though <00:02:49.079>we <00:02:49.219>had <00:02:49.359>that

00:02:50.750 --> 00:02:50.760 align:start position:0%
<00:02:50.750>And <00:02:50.751>still, <00:02:50.753>even <00:02:50.754>though <00:02:50.756>we <00:02:50.757>had <00:02:50.759>that

00:02:50.760 --> 00:02:53.630 align:start position:0%
<00:02:50.760>And <00:02:50.915>still, <00:02:51.071>even <00:02:51.226>though <00:02:51.382>we <00:02:51.537>had <00:02:51.693>that <00:02:51.848>on <00:02:52.004>January <00:02:52.160>2025,

00:02:53.630 --> 00:02:53.640 align:start position:0%
<00:02:53.630>on <00:02:53.634>January <00:02:53.638>2025,

00:02:53.640 --> 00:02:56.510 align:start position:0%
<00:02:53.640>on <00:02:53.779>January <00:02:53.920>2025, <00:02:54.059>as <00:02:54.200>you <00:02:54.339>can <00:02:54.480>see, <00:02:54.619>we <00:02:54.760>started <00:02:54.899>internal <00:02:55.040>beta

00:02:56.510 --> 00:02:56.520 align:start position:0%
<00:02:56.510>as <00:02:56.511>you <00:02:56.512>can <00:02:56.513>see, <00:02:56.515>we <00:02:56.516>started <00:02:56.517>internal <00:02:56.519>beta

00:02:56.520 --> 00:02:59.110 align:start position:0%
<00:02:56.520>as <00:02:56.620>you <00:02:56.720>can <00:02:56.820>see, <00:02:56.920>we <00:02:57.020>started <00:02:57.120>internal <00:02:57.220>beta <00:02:57.320>at <00:02:57.420>the <00:02:57.520>our <00:02:57.620>company <00:02:57.720>only <00:02:57.820>on <00:02:57.920>July.

00:02:59.110 --> 00:02:59.120 align:start position:0%
<00:02:59.110>at <00:02:59.111>the <00:02:59.113>our <00:02:59.114>company <00:02:59.116>only <00:02:59.117>on <00:02:59.119>July.

00:02:59.120 --> 00:03:01.430 align:start position:0%
<00:02:59.120>at <00:02:59.275>the <00:02:59.431>our <00:02:59.586>company <00:02:59.742>only <00:02:59.897>on <00:03:00.053>July. <00:03:00.208>So, <00:03:00.364>6 <00:03:00.520>month.

00:03:01.430 --> 00:03:01.440 align:start position:0%
<00:03:01.430>So, <00:03:01.434>6 <00:03:01.439>month.

00:03:01.440 --> 00:03:03.509 align:start position:0%
<00:03:01.440>So, <00:03:01.595>6 <00:03:01.751>month. <00:03:01.906>And <00:03:02.062>then <00:03:02.217>the <00:03:02.373>month <00:03:02.528>after, <00:03:02.684>August, <00:03:02.840>we

00:03:03.509 --> 00:03:03.519 align:start position:0%
<00:03:03.509>And <00:03:03.510>then <00:03:03.512>the <00:03:03.513>month <00:03:03.515>after, <00:03:03.516>August, <00:03:03.518>we

00:03:03.519 --> 00:03:05.030 align:start position:0%
<00:03:03.519>And <00:03:03.642>then <00:03:03.766>the <00:03:03.889>month <00:03:04.013>after, <00:03:04.137>August, <00:03:04.260>we <00:03:04.384>started <00:03:04.508>with <00:03:04.631>design <00:03:04.755>partners, <00:03:04.878>real

00:03:05.030 --> 00:03:05.040 align:start position:0%
<00:03:05.030>started <00:03:05.032>with <00:03:05.034>design <00:03:05.036>partners, <00:03:05.039>real

00:03:05.040 --> 00:03:07.830 align:start position:0%
<00:03:05.040>started <00:03:05.156>with <00:03:05.273>design <00:03:05.390>partners, <00:03:05.506>real <00:03:05.623>customers <00:03:05.739>who <00:03:05.856>work <00:03:05.973>with <00:03:06.090>us, <00:03:06.206>who <00:03:06.323>see <00:03:06.440>us

00:03:07.830 --> 00:03:07.840 align:start position:0%
<00:03:07.830>customers <00:03:07.831>who <00:03:07.832>work <00:03:07.833>with <00:03:07.835>us, <00:03:07.836>who <00:03:07.837>see <00:03:07.839>us

00:03:07.840 --> 00:03:10.830 align:start position:0%
<00:03:07.840>customers <00:03:07.980>who <00:03:08.120>work <00:03:08.260>with <00:03:08.400>us, <00:03:08.540>who <00:03:08.680>see <00:03:08.820>us <00:03:08.960>technical <00:03:09.100>um <00:03:09.240>partners.

00:03:10.830 --> 00:03:10.840 align:start position:0%
<00:03:10.830>technical <00:03:10.834>um <00:03:10.839>partners.

00:03:10.840 --> 00:03:12.229 align:start position:0%
<00:03:10.840>technical <00:03:10.965>um <00:03:11.090>partners. <00:03:11.215>And <00:03:11.340>I <00:03:11.465>think <00:03:11.590>this <00:03:11.715>is <00:03:11.840>where <00:03:11.965>things <00:03:12.090>started

00:03:12.229 --> 00:03:12.239 align:start position:0%
<00:03:12.228>And <00:03:12.230>I <00:03:12.231>think <00:03:12.232>this <00:03:12.234>is <00:03:12.235>where <00:03:12.236>things <00:03:12.238>started

00:03:12.239 --> 00:03:15.910 align:start position:0%
<00:03:12.239>And <00:03:12.339>I <00:03:12.439>think <00:03:12.539>this <00:03:12.639>is <00:03:12.739>where <00:03:12.839>things <00:03:12.939>started <00:03:13.039>to <00:03:13.139>get <00:03:13.239>interesting. <00:03:13.339>So, <00:03:13.439>until <00:03:13.539>that <00:03:13.639>part,

00:03:15.910 --> 00:03:15.920 align:start position:0%
<00:03:15.910>to <00:03:15.911>get <00:03:15.912>interesting. <00:03:15.914>So, <00:03:15.916>until <00:03:15.917>that <00:03:15.918>part,

00:03:15.920 --> 00:03:17.590 align:start position:0%
<00:03:15.920>to <00:03:16.036>get <00:03:16.153>interesting. <00:03:16.269>So, <00:03:16.386>until <00:03:16.503>that <00:03:16.619>part, <00:03:16.736>everything <00:03:16.853>was <00:03:16.970>going <00:03:17.086>extremely <00:03:17.203>well, <00:03:17.320>you

00:03:17.590 --> 00:03:17.600 align:start position:0%
<00:03:17.590>everything <00:03:17.591>was <00:03:17.593>going <00:03:17.595>extremely <00:03:17.597>well, <00:03:17.599>you

00:03:17.600 --> 00:03:19.910 align:start position:0%
<00:03:17.600>everything <00:03:17.716>was <00:03:17.833>going <00:03:17.950>extremely <00:03:18.066>well, <00:03:18.183>you <00:03:18.299>know, <00:03:18.416>it <00:03:18.533>Ollie <00:03:18.650>was <00:03:18.766>very <00:03:18.883>very <00:03:19.000>impressive.

00:03:19.910 --> 00:03:19.920 align:start position:0%
<00:03:19.910>know, <00:03:19.911>it <00:03:19.913>Ollie <00:03:19.914>was <00:03:19.916>very <00:03:19.917>very <00:03:19.919>impressive.

00:03:19.920 --> 00:03:22.030 align:start position:0%
<00:03:19.920>know, <00:03:20.027>it <00:03:20.135>Ollie <00:03:20.243>was <00:03:20.350>very <00:03:20.458>very <00:03:20.566>impressive. <00:03:20.673>Obviously, <00:03:20.781>the <00:03:20.889>demo <00:03:20.996>was <00:03:21.104>kind <00:03:21.212>of <00:03:21.320>shook

00:03:22.030 --> 00:03:22.040 align:start position:0%
<00:03:22.030>Obviously, <00:03:22.031>the <00:03:22.033>demo <00:03:22.034>was <00:03:22.036>kind <00:03:22.037>of <00:03:22.039>shook

00:03:22.040 --> 00:03:23.110 align:start position:0%
<00:03:22.040>Obviously, <00:03:22.177>the <00:03:22.315>demo <00:03:22.452>was <00:03:22.590>kind <00:03:22.727>of <00:03:22.865>shook <00:03:23.003>everyone.

00:03:23.110 --> 00:03:23.120 align:start position:0%
everyone.

00:03:23.120 --> 00:03:24.310 align:start position:0%
<00:03:23.120>everyone. <00:03:24.191>Um

00:03:24.310 --> 00:03:24.320 align:start position:0%
Um

00:03:24.320 --> 00:03:26.509 align:start position:0%
<00:03:24.320>Um <00:03:24.495>and <00:03:24.670>after <00:03:24.845>we <00:03:25.019>rolled <00:03:25.195>out <00:03:25.370>to <00:03:25.545>our <00:03:25.720>design

00:03:26.509 --> 00:03:26.519 align:start position:0%
<00:03:26.509>and <00:03:26.510>after <00:03:26.511>we <00:03:26.512>rolled <00:03:26.514>out <00:03:26.515>to <00:03:26.516>our <00:03:26.518>design

00:03:26.519 --> 00:03:27.870 align:start position:0%
<00:03:26.519>and <00:03:26.605>after <00:03:26.692>we <00:03:26.779>rolled <00:03:26.866>out <00:03:26.953>to <00:03:27.040>our <00:03:27.126>design <00:03:27.213>partners, <00:03:27.300>we <00:03:27.387>had <00:03:27.474>a <00:03:27.561>couple <00:03:27.648>of <00:03:27.734>design

00:03:27.870 --> 00:03:27.880 align:start position:0%
<00:03:27.870>partners, <00:03:27.871>we <00:03:27.873>had <00:03:27.874>a <00:03:27.876>couple <00:03:27.877>of <00:03:27.879>design

00:03:27.880 --> 00:03:30.229 align:start position:0%
<00:03:27.880>partners, <00:03:28.019>we <00:03:28.160>had <00:03:28.299>a <00:03:28.440>couple <00:03:28.579>of <00:03:28.720>design <00:03:28.859>partners <00:03:29.000>we <00:03:29.140>worked <00:03:29.280>with.

00:03:30.229 --> 00:03:30.239 align:start position:0%
<00:03:30.228>partners <00:03:30.232>we <00:03:30.234>worked <00:03:30.238>with.

00:03:30.239 --> 00:03:32.270 align:start position:0%
<00:03:30.239>partners <00:03:30.379>we <00:03:30.519>worked <00:03:30.659>with. <00:03:30.799>We <00:03:30.939>wanted <00:03:31.079>to <00:03:31.219>upgrade <00:03:31.359>the <00:03:31.499>model <00:03:31.639>version,

00:03:32.270 --> 00:03:32.280 align:start position:0%
<00:03:32.270>We <00:03:32.271>wanted <00:03:32.273>to <00:03:32.274>upgrade <00:03:32.276>the <00:03:32.277>model <00:03:32.279>version,

00:03:32.280 --> 00:03:34.229 align:start position:0%
<00:03:32.280>We <00:03:32.387>wanted <00:03:32.495>to <00:03:32.603>upgrade <00:03:32.710>the <00:03:32.818>model <00:03:32.926>version, <00:03:33.033>right? <00:03:33.141>So, <00:03:33.249>for <00:03:33.356>example, <00:03:33.464>we <00:03:33.572>updated <00:03:33.680>from

00:03:34.229 --> 00:03:34.239 align:start position:0%
<00:03:34.228>right? <00:03:34.230>So, <00:03:34.232>for <00:03:34.233>example, <00:03:34.234>we <00:03:34.236>updated <00:03:34.238>from

00:03:34.239 --> 00:03:37.509 align:start position:0%
<00:03:34.239>right? <00:03:34.339>So, <00:03:34.439>for <00:03:34.539>example, <00:03:34.639>we <00:03:34.739>updated <00:03:34.839>from <00:03:34.939>Sonnet <00:03:35.039>4.5 <00:03:35.139>to <00:03:35.239>4.6. <00:03:35.339>What <00:03:35.439>would <00:03:35.539>you <00:03:35.639>expect

00:03:37.509 --> 00:03:37.519 align:start position:0%
<00:03:37.509>Sonnet <00:03:37.510>4.5 <00:03:37.511>to <00:03:37.512>4.6. <00:03:37.514>What <00:03:37.515>would <00:03:37.516>you <00:03:37.518>expect

00:03:37.519 --> 00:03:38.870 align:start position:0%
<00:03:37.519>Sonnet <00:03:37.654>4.5 <00:03:37.789>to <00:03:37.924>4.6. <00:03:38.059>What <00:03:38.194>would <00:03:38.329>you <00:03:38.464>expect <00:03:38.599>to <00:03:38.734>happen?

00:03:38.870 --> 00:03:38.880 align:start position:0%
<00:03:38.870>to <00:03:38.879>happen?

00:03:38.880 --> 00:03:39.990 align:start position:0%
<00:03:38.880>to <00:03:39.379>happen? <00:03:39.879>Right?

00:03:39.990 --> 00:03:40.000 align:start position:0%
Right?

00:03:40.000 --> 00:03:42.190 align:start position:0%
<00:03:40.000>Right? <00:03:40.700>To <00:03:41.400>improve.

00:03:42.190 --> 00:03:42.200 align:start position:0%
<00:03:42.190>To <00:03:42.198>improve.

00:03:42.200 --> 00:03:43.350 align:start position:0%
<00:03:42.200>To <00:03:42.545>improve. <00:03:42.890>The <00:03:43.234>model

00:03:43.350 --> 00:03:43.360 align:start position:0%
<00:03:43.350>The <00:03:43.359>model

00:03:43.360 --> 00:03:44.710 align:start position:0%
<00:03:43.360>The <00:03:43.533>model <00:03:43.707>should <00:03:43.880>be <00:03:44.054>better, <00:03:44.227>right? <00:03:44.401>On <00:03:44.575>all

00:03:44.710 --> 00:03:44.720 align:start position:0%
<00:03:44.710>should <00:03:44.711>be <00:03:44.713>better, <00:03:44.715>right? <00:03:44.717>On <00:03:44.719>all

00:03:44.720 --> 00:03:46.350 align:start position:0%
<00:03:44.720>should <00:03:44.920>be <00:03:45.120>better, <00:03:45.320>right? <00:03:45.520>On <00:03:45.720>all <00:03:45.920>benchmarks, <00:03:46.120>we

00:03:46.350 --> 00:03:46.360 align:start position:0%
<00:03:46.350>benchmarks, <00:03:46.359>we

00:03:46.360 --> 00:03:49.310 align:start position:0%
<00:03:46.360>benchmarks, <00:03:46.500>we <00:03:46.640>uh <00:03:46.780>and <00:03:46.920>and <00:03:47.060>so <00:03:47.200>on <00:03:47.340>is <00:03:47.480>is <00:03:47.620>better. <00:03:47.760>So,

00:03:49.310 --> 00:03:49.320 align:start position:0%
<00:03:49.310>uh <00:03:49.311>and <00:03:49.312>and <00:03:49.313>so <00:03:49.314>on <00:03:49.315>is <00:03:49.316>is <00:03:49.317>better. <00:03:49.319>So,

00:03:49.320 --> 00:03:51.310 align:start position:0%
<00:03:49.320>uh <00:03:49.413>and <00:03:49.506>and <00:03:49.600>so <00:03:49.693>on <00:03:49.786>is <00:03:49.880>is <00:03:49.973>better. <00:03:50.066>So, <00:03:50.160>what? <00:03:50.253>Like <00:03:50.346>if <00:03:50.440>we <00:03:50.533>upgrade <00:03:50.626>the <00:03:50.720>model,

00:03:51.310 --> 00:03:51.320 align:start position:0%
<00:03:51.310>what? <00:03:51.311>Like <00:03:51.313>if <00:03:51.314>we <00:03:51.316>upgrade <00:03:51.317>the <00:03:51.319>model,

00:03:51.320 --> 00:03:53.870 align:start position:0%
<00:03:51.320>what? <00:03:51.447>Like <00:03:51.574>if <00:03:51.701>we <00:03:51.829>upgrade <00:03:51.956>the <00:03:52.083>model, <00:03:52.210>Ollie <00:03:52.338>should <00:03:52.465>now <00:03:52.592>be <00:03:52.720>better.

00:03:53.870 --> 00:03:53.880 align:start position:0%
<00:03:53.870>Ollie <00:03:53.872>should <00:03:53.874>now <00:03:53.876>be <00:03:53.879>better.

00:03:53.880 --> 00:03:56.470 align:start position:0%
<00:03:53.880>Ollie <00:03:54.035>should <00:03:54.191>now <00:03:54.346>be <00:03:54.502>better. <00:03:54.657>So, <00:03:54.813>the <00:03:54.968>funny <00:03:55.124>thing <00:03:55.280>is

00:03:56.470 --> 00:03:56.480 align:start position:0%
<00:03:56.470>So, <00:03:56.472>the <00:03:56.474>funny <00:03:56.476>thing <00:03:56.478>is

00:03:56.480 --> 00:03:57.910 align:start position:0%
<00:03:56.480>So, <00:03:56.737>the <00:03:56.994>funny <00:03:57.252>thing <00:03:57.509>is <00:03:57.767>um

00:03:57.910 --> 00:03:57.920 align:start position:0%
um

00:03:57.920 --> 00:03:59.430 align:start position:0%
<00:03:57.920>um <00:03:58.114>the <00:03:58.308>model <00:03:58.502>got <00:03:58.696>better, <00:03:58.890>but <00:03:59.084>our <00:03:59.279>agent

00:03:59.430 --> 00:03:59.440 align:start position:0%
<00:03:59.430>the <00:03:59.431>model <00:03:59.433>got <00:03:59.434>better, <00:03:59.436>but <00:03:59.437>our <00:03:59.439>agent

00:03:59.440 --> 00:04:00.870 align:start position:0%
<00:03:59.440>the <00:03:59.547>model <00:03:59.654>got <00:03:59.761>better, <00:03:59.869>but <00:03:59.976>our <00:04:00.083>agent <00:04:00.190>actually <00:04:00.298>got <00:04:00.405>worse. <00:04:00.512>So, <00:04:00.619>we <00:04:00.727>started

00:04:00.870 --> 00:04:00.880 align:start position:0%
<00:04:00.870>actually <00:04:00.871>got <00:04:00.873>worse. <00:04:00.875>So, <00:04:00.877>we <00:04:00.879>started

00:04:00.880 --> 00:04:02.350 align:start position:0%
<00:04:00.880>actually <00:04:01.012>got <00:04:01.144>worse. <00:04:01.276>So, <00:04:01.409>we <00:04:01.541>started <00:04:01.673>actually <00:04:01.806>getting <00:04:01.938>complaints <00:04:02.070>from <00:04:02.203>our

00:04:02.350 --> 00:04:02.360 align:start position:0%
<00:04:02.350>actually <00:04:02.352>getting <00:04:02.354>complaints <00:04:02.356>from <00:04:02.359>our

00:04:02.360 --> 00:04:04.430 align:start position:0%
<00:04:02.360>actually <00:04:02.535>getting <00:04:02.710>complaints <00:04:02.885>from <00:04:03.060>our <00:04:03.235>desired <00:04:03.410>smart <00:04:03.585>partners <00:04:03.760>saying,

00:04:04.430 --> 00:04:04.440 align:start position:0%
<00:04:04.430>desired <00:04:04.433>smart <00:04:04.436>partners <00:04:04.439>saying,

00:04:04.440 --> 00:04:06.670 align:start position:0%
<00:04:04.440>desired <00:04:04.556>smart <00:04:04.673>partners <00:04:04.790>saying, <00:04:04.906>"Hey <00:04:05.023>guys, <00:04:05.140>like <00:04:05.256>what's <00:04:05.373>going <00:04:05.490>on? <00:04:05.606>Uh <00:04:05.723>I <00:04:05.840>I

00:04:06.670 --> 00:04:06.680 align:start position:0%
<00:04:06.670>"Hey <00:04:06.671>guys, <00:04:06.672>like <00:04:06.673>what's <00:04:06.674>going <00:04:06.675>on? <00:04:06.676>Uh <00:04:06.677>I <00:04:06.679>I

00:04:06.680 --> 00:04:08.830 align:start position:0%
<00:04:06.680>"Hey <00:04:06.767>guys, <00:04:06.855>like <00:04:06.942>what's <00:04:07.030>going <00:04:07.117>on? <00:04:07.205>Uh <00:04:07.292>I <00:04:07.380>I <00:04:07.467>feel <00:04:07.555>like <00:04:07.642>Ollie <00:04:07.730>is <00:04:07.817>not <00:04:07.905>the <00:04:07.992>same <00:04:08.080>Ollie

00:04:08.830 --> 00:04:08.840 align:start position:0%
<00:04:08.830>feel <00:04:08.831>like <00:04:08.832>Ollie <00:04:08.833>is <00:04:08.835>not <00:04:08.836>the <00:04:08.837>same <00:04:08.839>Ollie

00:04:08.840 --> 00:04:10.670 align:start position:0%
<00:04:08.840>feel <00:04:08.956>like <00:04:09.073>Ollie <00:04:09.190>is <00:04:09.306>not <00:04:09.423>the <00:04:09.540>same <00:04:09.656>Ollie <00:04:09.773>that <00:04:09.890>we <00:04:10.006>had <00:04:10.123>last <00:04:10.240>week.

00:04:10.670 --> 00:04:10.680 align:start position:0%
<00:04:10.670>that <00:04:10.672>we <00:04:10.674>had <00:04:10.676>last <00:04:10.679>week.

00:04:10.680 --> 00:04:13.270 align:start position:0%
<00:04:10.680>that <00:04:10.787>we <00:04:10.895>had <00:04:11.003>last <00:04:11.110>week. <00:04:11.218>Um <00:04:11.326>what <00:04:11.433>have <00:04:11.541>you <00:04:11.649>changed? <00:04:11.756>I <00:04:11.864>I <00:04:11.972>feel <00:04:12.080>like

00:04:13.270 --> 00:04:13.280 align:start position:0%
<00:04:13.270>Um <00:04:13.271>what <00:04:13.272>have <00:04:13.273>you <00:04:13.274>changed? <00:04:13.275>I <00:04:13.276>I <00:04:13.277>feel <00:04:13.279>like

00:04:13.280 --> 00:04:16.470 align:start position:0%
<00:04:13.280>Um <00:04:13.373>what <00:04:13.466>have <00:04:13.560>you <00:04:13.653>changed? <00:04:13.746>I <00:04:13.840>I <00:04:13.933>feel <00:04:14.026>like <00:04:14.120>Ollie <00:04:14.213>is <00:04:14.306>not <00:04:14.400>providing <00:04:14.493>the <00:04:14.586>exact <00:04:14.680>answers

00:04:16.470 --> 00:04:16.480 align:start position:0%
<00:04:16.470>Ollie <00:04:16.471>is <00:04:16.473>not <00:04:16.474>providing <00:04:16.476>the <00:04:16.477>exact <00:04:16.479>answers

00:04:16.480 --> 00:04:18.590 align:start position:0%
<00:04:16.480>Ollie <00:04:16.620>is <00:04:16.760>not <00:04:16.900>providing <00:04:17.040>the <00:04:17.180>exact <00:04:17.320>answers <00:04:17.460>like <00:04:17.600>it <00:04:17.740>used <00:04:17.880>to."

00:04:18.590 --> 00:04:18.600 align:start position:0%
<00:04:18.589>like <00:04:18.592>it <00:04:18.596>used <00:04:18.599>to."

00:04:18.600 --> 00:04:20.470 align:start position:0%
<00:04:18.600>like <00:04:18.727>it <00:04:18.854>used <00:04:18.981>to." <00:04:19.109>Um <00:04:19.236>so, <00:04:19.363>it <00:04:19.490>was <00:04:19.618>very <00:04:19.745>very <00:04:19.872>interesting <00:04:20.000>for

00:04:20.470 --> 00:04:20.480 align:start position:0%
<00:04:20.470>Um <00:04:20.471>so, <00:04:20.472>it <00:04:20.473>was <00:04:20.475>very <00:04:20.476>very <00:04:20.477>interesting <00:04:20.479>for

00:04:20.480 --> 00:04:24.270 align:start position:0%
<00:04:20.480>Um <00:04:20.587>so, <00:04:20.695>it <00:04:20.803>was <00:04:20.910>very <00:04:21.018>very <00:04:21.126>interesting <00:04:21.233>for <00:04:21.341>us. <00:04:21.449>And <00:04:21.556>what <00:04:21.664>we <00:04:21.772>realized <00:04:21.880>is

00:04:24.270 --> 00:04:24.280 align:start position:0%
<00:04:24.270>us. <00:04:24.271>And <00:04:24.273>what <00:04:24.275>we <00:04:24.277>realized <00:04:24.279>is

00:04:24.280 --> 00:04:26.710 align:start position:0%
<00:04:24.280>us. <00:04:24.407>And <00:04:24.534>what <00:04:24.661>we <00:04:24.789>realized <00:04:24.916>is <00:04:25.043>upgrading <00:04:25.170>a <00:04:25.298>model <00:04:25.425>completely <00:04:25.552>changes <00:04:25.679>the

00:04:26.710 --> 00:04:26.720 align:start position:0%
<00:04:26.710>upgrading <00:04:26.711>a <00:04:26.713>model <00:04:26.715>completely <00:04:26.717>changes <00:04:26.719>the

00:04:26.720 --> 00:04:28.750 align:start position:0%
<00:04:26.720>upgrading <00:04:26.836>a <00:04:26.953>model <00:04:27.070>completely <00:04:27.186>changes <00:04:27.303>the <00:04:27.420>behavior <00:04:27.536>of <00:04:27.653>your <00:04:27.770>agent. <00:04:27.886>Okay, <00:04:28.003>so <00:04:28.120>you

00:04:28.750 --> 00:04:28.760 align:start position:0%
<00:04:28.750>behavior <00:04:28.751>of <00:04:28.753>your <00:04:28.754>agent. <00:04:28.756>Okay, <00:04:28.757>so <00:04:28.759>you

00:04:28.760 --> 00:04:30.430 align:start position:0%
<00:04:28.760>behavior <00:04:28.867>of <00:04:28.975>your <00:04:29.083>agent. <00:04:29.190>Okay, <00:04:29.298>so <00:04:29.406>you <00:04:29.513>cannot <00:04:29.621>just <00:04:29.729>do <00:04:29.836>it. <00:04:29.944>It's <00:04:30.052>not <00:04:30.159>just

00:04:30.430 --> 00:04:30.440 align:start position:0%
<00:04:30.430>cannot <00:04:30.431>just <00:04:30.433>do <00:04:30.434>it. <00:04:30.436>It's <00:04:30.437>not <00:04:30.439>just

00:04:30.440 --> 00:04:34.030 align:start position:0%
<00:04:30.440>cannot <00:04:30.580>just <00:04:30.719>do <00:04:30.860>it. <00:04:31.000>It's <00:04:31.140>not <00:04:31.280>just <00:04:31.420>updating <00:04:31.560>one <00:04:31.700>single <00:04:31.840>string.

00:04:34.030 --> 00:04:34.040 align:start position:0%
<00:04:34.030>updating <00:04:34.033>one <00:04:34.036>single <00:04:34.039>string.

00:04:34.040 --> 00:04:36.710 align:start position:0%
<00:04:34.040>updating <00:04:34.240>one <00:04:34.440>single <00:04:34.640>string. <00:04:34.840>Um <00:04:35.040>so, <00:04:35.240>for <00:04:35.440>that

00:04:36.710 --> 00:04:36.720 align:start position:0%
<00:04:36.710>Um <00:04:36.712>so, <00:04:36.716>for <00:04:36.719>that

00:04:36.720 --> 00:04:39.670 align:start position:0%
<00:04:36.720>Um <00:04:36.836>so, <00:04:36.953>for <00:04:37.070>that <00:04:37.186>what <00:04:37.303>we <00:04:37.420>have <00:04:37.536>realized <00:04:37.653>we <00:04:37.770>need <00:04:37.886>to <00:04:38.003>do <00:04:38.120>is

00:04:39.670 --> 00:04:39.680 align:start position:0%
<00:04:39.670>what <00:04:39.671>we <00:04:39.672>have <00:04:39.673>realized <00:04:39.674>we <00:04:39.675>need <00:04:39.676>to <00:04:39.677>do <00:04:39.679>is

00:04:39.680 --> 00:04:42.310 align:start position:0%
<00:04:39.680>what <00:04:39.796>we <00:04:39.913>have <00:04:40.030>realized <00:04:40.146>we <00:04:40.263>need <00:04:40.380>to <00:04:40.496>do <00:04:40.613>is <00:04:40.730>for <00:04:40.846>every <00:04:40.963>model <00:04:41.080>change,

00:04:42.310 --> 00:04:42.320 align:start position:0%
<00:04:42.310>for <00:04:42.313>every <00:04:42.316>model <00:04:42.319>change,

00:04:42.320 --> 00:04:43.270 align:start position:0%
<00:04:42.320>for <00:04:42.533>every <00:04:42.747>model <00:04:42.961>change, <00:04:43.174>okay?

00:04:43.270 --> 00:04:43.280 align:start position:0%
okay?

00:04:43.280 --> 00:04:45.350 align:start position:0%
<00:04:43.280>okay? <00:04:43.513>It's <00:04:43.746>not <00:04:43.979>just <00:04:44.213>a <00:04:44.446>string <00:04:44.679>update.

00:04:45.350 --> 00:04:45.360 align:start position:0%
<00:04:45.350>It's <00:04:45.351>not <00:04:45.353>just <00:04:45.355>a <00:04:45.357>string <00:04:45.359>update.

00:04:45.360 --> 00:04:47.510 align:start position:0%
<00:04:45.360>It's <00:04:45.467>not <00:04:45.575>just <00:04:45.683>a <00:04:45.790>string <00:04:45.898>update. <00:04:46.006>We <00:04:46.113>treat <00:04:46.221>this <00:04:46.329>as <00:04:46.436>a <00:04:46.544>migration, <00:04:46.652>okay? <00:04:46.760>Not

00:04:47.510 --> 00:04:47.520 align:start position:0%
<00:04:47.510>We <00:04:47.511>treat <00:04:47.512>this <00:04:47.513>as <00:04:47.515>a <00:04:47.516>migration, <00:04:47.517>okay? <00:04:47.519>Not

00:04:47.520 --> 00:04:49.070 align:start position:0%
<00:04:47.520>We <00:04:47.613>treat <00:04:47.705>this <00:04:47.799>as <00:04:47.892>a <00:04:47.985>migration, <00:04:48.078>okay? <00:04:48.171>Not <00:04:48.264>just <00:04:48.357>as <00:04:48.450>a <00:04:48.543>toggle. <00:04:48.635>And <00:04:48.729>this <00:04:48.822>is <00:04:48.914>really

00:04:49.070 --> 00:04:49.080 align:start position:0%
<00:04:49.070>just <00:04:49.071>as <00:04:49.072>a <00:04:49.073>toggle. <00:04:49.075>And <00:04:49.076>this <00:04:49.077>is <00:04:49.079>really

00:04:49.080 --> 00:04:51.110 align:start position:0%
<00:04:49.080>just <00:04:49.187>as <00:04:49.295>a <00:04:49.403>toggle. <00:04:49.510>And <00:04:49.618>this <00:04:49.726>is <00:04:49.833>really <00:04:49.941>really <00:04:50.049>important. <00:04:50.156>When <00:04:50.264>you <00:04:50.372>change <00:04:50.479>a

00:04:51.110 --> 00:04:51.120 align:start position:0%
<00:04:51.110>really <00:04:51.111>important. <00:04:51.113>When <00:04:51.115>you <00:04:51.117>change <00:04:51.119>a

00:04:51.120 --> 00:04:53.270 align:start position:0%
<00:04:51.120>really <00:04:51.275>important. <00:04:51.431>When <00:04:51.586>you <00:04:51.742>change <00:04:51.897>a <00:04:52.053>model, <00:04:52.208>you <00:04:52.364>need <00:04:52.520>to

00:04:53.270 --> 00:04:53.280 align:start position:0%
<00:04:53.270>model, <00:04:53.272>you <00:04:53.275>need <00:04:53.279>to

00:04:53.280 --> 00:04:55.190 align:start position:0%
<00:04:53.280>model, <00:04:53.435>you <00:04:53.591>need <00:04:53.746>to <00:04:53.902>fine-tune <00:04:54.057>and <00:04:54.213>then <00:04:54.368>change <00:04:54.524>your <00:04:54.679>system

00:04:55.190 --> 00:04:55.200 align:start position:0%
<00:04:55.190>fine-tune <00:04:55.191>and <00:04:55.193>then <00:04:55.195>change <00:04:55.197>your <00:04:55.199>system

00:04:55.200 --> 00:04:56.110 align:start position:0%
<00:04:55.200>fine-tune <00:04:55.336>and <00:04:55.473>then <00:04:55.609>change <00:04:55.746>your <00:04:55.882>system <00:04:56.019>prompt.

00:04:56.110 --> 00:04:56.120 align:start position:0%
prompt.

00:04:56.120 --> 00:04:58.150 align:start position:0%
<00:04:56.120>prompt. <00:04:56.295>You <00:04:56.470>need <00:04:56.645>to <00:04:56.820>adapt <00:04:56.995>the <00:04:57.170>description <00:04:57.345>of <00:04:57.520>the

00:04:58.150 --> 00:04:58.160 align:start position:0%
<00:04:58.150>You <00:04:58.151>need <00:04:58.152>to <00:04:58.153>adapt <00:04:58.155>the <00:04:58.156>description <00:04:58.157>of <00:04:58.159>the

00:04:58.160 --> 00:05:00.390 align:start position:0%
<00:04:58.159>You <00:04:58.267>need <00:04:58.375>to <00:04:58.483>adapt <00:04:58.590>the <00:04:58.698>description <00:04:58.806>of <00:04:58.913>the <00:04:59.021>tool <00:04:59.129>calls. <00:04:59.236>Why? <00:04:59.344>Because <00:04:59.452>every <00:04:59.559>model,

00:05:00.390 --> 00:05:00.400 align:start position:0%
<00:05:00.390>tool <00:05:00.391>calls. <00:05:00.393>Why? <00:05:00.395>Because <00:05:00.397>every <00:05:00.399>model,

00:05:00.400 --> 00:05:02.790 align:start position:0%
<00:05:00.400>tool <00:05:00.516>calls. <00:05:00.633>Why? <00:05:00.750>Because <00:05:00.866>every <00:05:00.983>model, <00:05:01.099>even <00:05:01.216>if <00:05:01.333>it's <00:05:01.450>from <00:05:01.566>the <00:05:01.683>same <00:05:01.799>family,

00:05:02.790 --> 00:05:02.800 align:start position:0%
<00:05:02.790>even <00:05:02.791>if <00:05:02.793>it's <00:05:02.794>from <00:05:02.796>the <00:05:02.797>same <00:05:02.799>family,

00:05:02.800 --> 00:05:04.670 align:start position:0%
<00:05:02.800>even <00:05:02.927>if <00:05:03.054>it's <00:05:03.181>from <00:05:03.309>the <00:05:03.436>same <00:05:03.563>family, <00:05:03.690>have <00:05:03.818>unique <00:05:03.945>characteristics. <00:05:04.072>And <00:05:04.200>in

00:05:04.670 --> 00:05:04.680 align:start position:0%
<00:05:04.670>have <00:05:04.672>unique <00:05:04.674>characteristics. <00:05:04.676>And <00:05:04.679>in

00:05:04.680 --> 00:05:07.270 align:start position:0%
<00:05:04.680>have <00:05:04.796>unique <00:05:04.913>characteristics. <00:05:05.030>And <00:05:05.146>in <00:05:05.263>order <00:05:05.380>to <00:05:05.496>achieve <00:05:05.613>and <00:05:05.730>for <00:05:05.846>the <00:05:05.963>very <00:05:06.080>least

00:05:07.270 --> 00:05:07.280 align:start position:0%
<00:05:07.270>order <00:05:07.271>to <00:05:07.272>achieve <00:05:07.273>and <00:05:07.275>for <00:05:07.276>the <00:05:07.277>very <00:05:07.279>least

00:05:07.280 --> 00:05:08.870 align:start position:0%
<00:05:07.280>order <00:05:07.387>to <00:05:07.495>achieve <00:05:07.603>and <00:05:07.710>for <00:05:07.818>the <00:05:07.926>very <00:05:08.033>least <00:05:08.141>maintain <00:05:08.249>the <00:05:08.356>quality, <00:05:08.464>if <00:05:08.572>not <00:05:08.679>even

00:05:08.870 --> 00:05:08.880 align:start position:0%
<00:05:08.870>maintain <00:05:08.871>the <00:05:08.873>quality, <00:05:08.875>if <00:05:08.877>not <00:05:08.879>even

00:05:08.880 --> 00:05:11.830 align:start position:0%
<00:05:08.880>maintain <00:05:09.007>the <00:05:09.134>quality, <00:05:09.261>if <00:05:09.389>not <00:05:09.516>even <00:05:09.643>improve, <00:05:09.770>that's <00:05:09.898>what <00:05:10.025>we <00:05:10.152>aim <00:05:10.280>for,

00:05:11.830 --> 00:05:11.840 align:start position:0%
<00:05:11.830>improve, <00:05:11.831>that's <00:05:11.833>what <00:05:11.835>we <00:05:11.837>aim <00:05:11.839>for,

00:05:11.840 --> 00:05:14.750 align:start position:0%
<00:05:11.840>improve, <00:05:11.995>that's <00:05:12.151>what <00:05:12.306>we <00:05:12.462>aim <00:05:12.617>for, <00:05:12.773>we <00:05:12.928>have <00:05:13.084>to <00:05:13.239>evaluate.

00:05:14.750 --> 00:05:14.760 align:start position:0%
<00:05:14.750>we <00:05:14.753>have <00:05:14.756>to <00:05:14.759>evaluate.

00:05:14.760 --> 00:05:16.390 align:start position:0%
<00:05:14.760>we <00:05:14.900>have <00:05:15.039>to <00:05:15.180>evaluate. <00:05:15.320>Okay, <00:05:15.460>so <00:05:15.599>we <00:05:15.740>have <00:05:15.880>our <00:05:16.020>own <00:05:16.159>evaluation

00:05:16.390 --> 00:05:16.400 align:start position:0%
<00:05:16.390>Okay, <00:05:16.391>so <00:05:16.393>we <00:05:16.394>have <00:05:16.395>our <00:05:16.397>own <00:05:16.399>evaluation

00:05:16.400 --> 00:05:18.510 align:start position:0%
<00:05:16.400>Okay, <00:05:16.507>so <00:05:16.615>we <00:05:16.723>have <00:05:16.830>our <00:05:16.938>own <00:05:17.046>evaluation <00:05:17.153>system, <00:05:17.261>which <00:05:17.369>I <00:05:17.476>can <00:05:17.584>talk <00:05:17.692>about <00:05:17.799>later.

00:05:18.510 --> 00:05:18.520 align:start position:0%
<00:05:18.510>system, <00:05:18.511>which <00:05:18.513>I <00:05:18.514>can <00:05:18.515>talk <00:05:18.517>about <00:05:18.519>later.

00:05:18.520 --> 00:05:20.710 align:start position:0%
<00:05:18.520>system, <00:05:18.613>which <00:05:18.706>I <00:05:18.799>can <00:05:18.893>talk <00:05:18.986>about <00:05:19.080>later. <00:05:19.173>So, <00:05:19.266>we <00:05:19.359>ran <00:05:19.453>our <00:05:19.546>all <00:05:19.640>We <00:05:19.733>We're <00:05:19.826>running <00:05:19.919>all

00:05:20.710 --> 00:05:20.720 align:start position:0%
<00:05:20.710>So, <00:05:20.711>we <00:05:20.712>ran <00:05:20.713>our <00:05:20.714>all <00:05:20.715>We <00:05:20.716>We're <00:05:20.717>running <00:05:20.719>all

00:05:20.720 --> 00:05:23.270 align:start position:0%
<00:05:20.720>So, <00:05:20.813>we <00:05:20.906>ran <00:05:21.000>our <00:05:21.093>all <00:05:21.186>We <00:05:21.280>We're <00:05:21.373>running <00:05:21.466>all <00:05:21.560>of <00:05:21.653>our <00:05:21.746>evaluations. <00:05:21.840>We <00:05:21.933>see <00:05:22.026>today <00:05:22.120>what's

00:05:23.270 --> 00:05:23.280 align:start position:0%
<00:05:23.270>of <00:05:23.271>our <00:05:23.272>evaluations. <00:05:23.274>We <00:05:23.275>see <00:05:23.277>today <00:05:23.279>what's

00:05:23.280 --> 00:05:25.110 align:start position:0%
<00:05:23.280>of <00:05:23.387>our <00:05:23.495>evaluations. <00:05:23.603>We <00:05:23.710>see <00:05:23.818>today <00:05:23.926>what's <00:05:24.033>working <00:05:24.141>well, <00:05:24.249>what <00:05:24.356>doesn't <00:05:24.464>with <00:05:24.572>the <00:05:24.679>new

00:05:25.110 --> 00:05:25.120 align:start position:0%
<00:05:25.110>working <00:05:25.111>well, <00:05:25.113>what <00:05:25.114>doesn't <00:05:25.116>with <00:05:25.117>the <00:05:25.119>new

00:05:25.120 --> 00:05:27.950 align:start position:0%
<00:05:25.120>working <00:05:25.227>well, <00:05:25.335>what <00:05:25.443>doesn't <00:05:25.550>with <00:05:25.658>the <00:05:25.766>new <00:05:25.873>model. <00:05:25.981>Then <00:05:26.089>we <00:05:26.196>make <00:05:26.304>the <00:05:26.412>adaptations, <00:05:26.520>and

00:05:27.950 --> 00:05:27.960 align:start position:0%
<00:05:27.950>model. <00:05:27.951>Then <00:05:27.953>we <00:05:27.954>make <00:05:27.955>the <00:05:27.957>adaptations, <00:05:27.959>and

00:05:27.960 --> 00:05:30.950 align:start position:0%
<00:05:27.960>model. <00:05:28.060>Then <00:05:28.159>we <00:05:28.260>make <00:05:28.359>the <00:05:28.460>adaptations, <00:05:28.560>and <00:05:28.659>only <00:05:28.760>after <00:05:28.859>we <00:05:28.960>get <00:05:29.060>to <00:05:29.159>the <00:05:29.260>conviction <00:05:29.359>that

00:05:30.950 --> 00:05:30.960 align:start position:0%
<00:05:30.950>only <00:05:30.951>after <00:05:30.952>we <00:05:30.953>get <00:05:30.955>to <00:05:30.956>the <00:05:30.957>conviction <00:05:30.959>that

00:05:30.960 --> 00:05:35.470 align:start position:0%
<00:05:30.960>only <00:05:31.060>after <00:05:31.159>we <00:05:31.260>get <00:05:31.359>to <00:05:31.460>the <00:05:31.560>conviction <00:05:31.659>that <00:05:31.760>really <00:05:31.859>upgrading <00:05:31.960>from <00:05:32.060>Sonat <00:05:32.159>4.5 <00:05:32.260>to <00:05:32.359>4.6,

00:05:35.470 --> 00:05:35.480 align:start position:0%
<00:05:35.470>really <00:05:35.471>upgrading <00:05:35.473>from <00:05:35.474>Sonat <00:05:35.476>4.5 <00:05:35.477>to <00:05:35.479>4.6,

00:05:35.480 --> 00:05:38.390 align:start position:0%
<00:05:35.480>really <00:05:35.607>upgrading <00:05:35.734>from <00:05:35.861>Sonat <00:05:35.989>4.5 <00:05:36.116>to <00:05:36.243>4.6, <00:05:36.370>we <00:05:36.498>actually <00:05:36.625>get <00:05:36.752>dramatic <00:05:36.880>improvement.

00:05:38.390 --> 00:05:38.400 align:start position:0%
<00:05:38.390>we <00:05:38.392>actually <00:05:38.394>get <00:05:38.396>dramatic <00:05:38.399>improvement.

00:05:38.400 --> 00:05:39.670 align:start position:0%
<00:05:38.400>we <00:05:38.487>actually <00:05:38.575>get <00:05:38.663>dramatic <00:05:38.751>improvement. <00:05:38.839>So, <00:05:38.927>I <00:05:39.015>think <00:05:39.103>this <00:05:39.191>is <00:05:39.279>one <00:05:39.367>of <00:05:39.455>an <00:05:39.543>an

00:05:39.670 --> 00:05:39.680 align:start position:0%
<00:05:39.670>So, <00:05:39.671>I <00:05:39.672>think <00:05:39.673>this <00:05:39.674>is <00:05:39.675>one <00:05:39.676>of <00:05:39.677>an <00:05:39.679>an

00:05:39.680 --> 00:05:42.230 align:start position:0%
<00:05:39.680>So, <00:05:39.807>I <00:05:39.934>think <00:05:40.061>this <00:05:40.189>is <00:05:40.316>one <00:05:40.443>of <00:05:40.570>an <00:05:40.698>an <00:05:40.825>interesting <00:05:40.952>takeaway. <00:05:41.080>Um

00:05:42.230 --> 00:05:42.240 align:start position:0%
<00:05:42.230>interesting <00:05:42.234>takeaway. <00:05:42.239>Um

00:05:42.240 --> 00:05:44.670 align:start position:0%
<00:05:42.240>interesting <00:05:42.356>takeaway. <00:05:42.473>Um <00:05:42.590>and <00:05:42.706>as <00:05:42.823>we <00:05:42.940>think <00:05:43.056>about <00:05:43.173>it <00:05:43.290>not <00:05:43.406>only <00:05:43.523>as <00:05:43.640>a

00:05:44.670 --> 00:05:44.680 align:start position:0%
<00:05:44.670>and <00:05:44.671>as <00:05:44.672>we <00:05:44.673>think <00:05:44.674>about <00:05:44.675>it <00:05:44.676>not <00:05:44.677>only <00:05:44.678>as <00:05:44.679>a

00:05:44.680 --> 00:05:46.590 align:start position:0%
<00:05:44.680>and <00:05:44.773>as <00:05:44.866>we <00:05:44.960>think <00:05:45.053>about <00:05:45.146>it <00:05:45.240>not <00:05:45.333>only <00:05:45.426>as <00:05:45.520>a <00:05:45.613>toggle, <00:05:45.706>but <00:05:45.800>actually <00:05:45.893>as <00:05:45.986>a <00:05:46.080>migration,

00:05:46.590 --> 00:05:46.600 align:start position:0%
<00:05:46.590>toggle, <00:05:46.591>but <00:05:46.593>actually <00:05:46.595>as <00:05:46.597>a <00:05:46.599>migration,

00:05:46.600 --> 00:05:47.830 align:start position:0%
<00:05:46.600>toggle, <00:05:46.692>but <00:05:46.784>actually <00:05:46.876>as <00:05:46.969>a <00:05:47.061>migration, <00:05:47.153>think <00:05:47.245>about <00:05:47.338>when <00:05:47.430>you <00:05:47.522>deploy <00:05:47.614>a <00:05:47.707>new

00:05:47.830 --> 00:05:47.840 align:start position:0%
<00:05:47.830>think <00:05:47.831>about <00:05:47.832>when <00:05:47.834>you <00:05:47.836>deploy <00:05:47.837>a <00:05:47.839>new

00:05:47.840 --> 00:05:49.190 align:start position:0%
<00:05:47.840>think <00:05:47.941>about <00:05:48.042>when <00:05:48.143>you <00:05:48.245>deploy <00:05:48.346>a <00:05:48.447>new <00:05:48.548>version <00:05:48.650>to <00:05:48.751>production, <00:05:48.852>right? <00:05:48.953>Like <00:05:49.055>you

00:05:49.190 --> 00:05:49.200 align:start position:0%
<00:05:49.190>version <00:05:49.191>to <00:05:49.193>production, <00:05:49.195>right? <00:05:49.197>Like <00:05:49.199>you

00:05:49.200 --> 00:05:50.950 align:start position:0%
<00:05:49.200>version <00:05:49.316>to <00:05:49.433>production, <00:05:49.550>right? <00:05:49.666>Like <00:05:49.783>you <00:05:49.900>don't <00:05:50.016>immediately <00:05:50.133>deploy <00:05:50.250>to <00:05:50.366>all <00:05:50.483>of <00:05:50.599>your

00:05:50.950 --> 00:05:50.960 align:start position:0%
<00:05:50.950>don't <00:05:50.951>immediately <00:05:50.953>deploy <00:05:50.954>to <00:05:50.955>all <00:05:50.957>of <00:05:50.959>your

00:05:50.960 --> 00:05:55.830 align:start position:0%
<00:05:50.960>don't <00:05:51.067>immediately <00:05:51.175>deploy <00:05:51.283>to <00:05:51.390>all <00:05:51.498>of <00:05:51.606>your <00:05:51.713>customers <00:05:51.821>unless <00:05:51.929>you <00:05:52.036>you <00:05:52.144>go <00:05:52.252>YOLO. <00:05:52.359>Um

00:05:55.830 --> 00:05:55.840 align:start position:0%
<00:05:55.830>customers <00:05:55.831>unless <00:05:55.832>you <00:05:55.834>you <00:05:55.836>go <00:05:55.837>YOLO. <00:05:55.839>Um

00:05:55.840 --> 00:05:59.830 align:start position:0%
<00:05:55.840>customers <00:05:55.956>unless <00:05:56.073>you <00:05:56.190>you <00:05:56.306>go <00:05:56.423>YOLO. <00:05:56.540>Um <00:05:56.656>and <00:05:56.773>therefore <00:05:56.890>same <00:05:57.006>thing <00:05:57.123>with <00:05:57.240>models.

00:05:59.830 --> 00:05:59.840 align:start position:0%
<00:05:59.830>and <00:05:59.831>therefore <00:05:59.833>same <00:05:59.835>thing <00:05:59.837>with <00:05:59.839>models.

00:05:59.840 --> 00:06:00.430 align:start position:0%
<00:05:59.840>and <00:05:59.928>therefore <00:06:00.017>same <00:06:00.105>thing <00:06:00.194>with <00:06:00.282>models. <00:06:00.371>Uh

00:06:00.430 --> 00:06:00.440 align:start position:0%
Uh

00:06:00.440 --> 00:06:02.310 align:start position:0%
<00:06:00.440>Uh <00:06:00.595>by <00:06:00.751>the <00:06:00.906>way, <00:06:01.062>it's <00:06:01.217>quite <00:06:01.373>funny <00:06:01.528>to <00:06:01.684>see <00:06:01.840>how

00:06:02.310 --> 00:06:02.320 align:start position:0%
<00:06:02.310>by <00:06:02.311>the <00:06:02.312>way, <00:06:02.313>it's <00:06:02.314>quite <00:06:02.315>funny <00:06:02.316>to <00:06:02.317>see <00:06:02.319>how

00:06:02.320 --> 00:06:04.630 align:start position:0%
<00:06:02.320>by <00:06:02.407>the <00:06:02.495>way, <00:06:02.582>it's <00:06:02.670>quite <00:06:02.757>funny <00:06:02.844>to <00:06:02.932>see <00:06:03.020>how <00:06:03.107>these <00:06:03.195>things <00:06:03.282>behave. <00:06:03.370>I <00:06:03.457>can <00:06:03.545>tell <00:06:03.632>you <00:06:03.719>when

00:06:04.630 --> 00:06:04.640 align:start position:0%
<00:06:04.630>these <00:06:04.631>things <00:06:04.632>behave. <00:06:04.633>I <00:06:04.635>can <00:06:04.636>tell <00:06:04.637>you <00:06:04.639>when

00:06:04.640 --> 00:06:05.630 align:start position:0%
<00:06:04.640>these <00:06:04.751>things <00:06:04.862>behave. <00:06:04.974>I <00:06:05.085>can <00:06:05.196>tell <00:06:05.308>you <00:06:05.419>when <00:06:05.531>we've

00:06:05.630 --> 00:06:05.640 align:start position:0%
we've

00:06:05.640 --> 00:06:07.550 align:start position:0%
<00:06:05.640>we've <00:06:05.873>allowed <00:06:06.106>our <00:06:06.340>customers <00:06:06.573>to <00:06:06.806>actually <00:06:07.039>choose

00:06:07.550 --> 00:06:07.560 align:start position:0%
<00:06:07.550>allowed <00:06:07.551>our <00:06:07.553>customers <00:06:07.555>to <00:06:07.557>actually <00:06:07.559>choose

00:06:07.560 --> 00:06:09.110 align:start position:0%
<00:06:07.560>allowed <00:06:07.699>our <00:06:07.839>customers <00:06:07.978>to <00:06:08.118>actually <00:06:08.257>choose <00:06:08.397>the <00:06:08.536>model <00:06:08.676>on <00:06:08.815>their <00:06:08.955>own.

00:06:09.110 --> 00:06:09.120 align:start position:0%
<00:06:09.110>the <00:06:09.112>model <00:06:09.114>on <00:06:09.116>their <00:06:09.119>own.

00:06:09.120 --> 00:06:10.870 align:start position:0%
<00:06:09.120>the <00:06:09.236>model <00:06:09.353>on <00:06:09.470>their <00:06:09.586>own. <00:06:09.703>So <00:06:09.820>for <00:06:09.936>example, <00:06:10.053>not <00:06:10.170>only <00:06:10.286>we <00:06:10.403>started <00:06:10.520>with

00:06:10.870 --> 00:06:10.880 align:start position:0%
<00:06:10.870>So <00:06:10.871>for <00:06:10.872>example, <00:06:10.873>not <00:06:10.875>only <00:06:10.876>we <00:06:10.877>started <00:06:10.879>with

00:06:10.880 --> 00:06:13.750 align:start position:0%
<00:06:10.880>So <00:06:10.967>for <00:06:11.055>example, <00:06:11.142>not <00:06:11.230>only <00:06:11.317>we <00:06:11.405>started <00:06:11.492>with <00:06:11.580>GPT <00:06:11.667>but <00:06:11.755>then <00:06:11.842>we <00:06:11.930>added <00:06:12.017>a <00:06:12.105>cloud <00:06:12.192>family <00:06:12.280>and

00:06:13.750 --> 00:06:13.760 align:start position:0%
<00:06:13.750>GPT <00:06:13.751>but <00:06:13.752>then <00:06:13.753>we <00:06:13.754>added <00:06:13.755>a <00:06:13.756>cloud <00:06:13.757>family <00:06:13.759>and

00:06:13.760 --> 00:06:15.310 align:start position:0%
<00:06:13.760>GPT <00:06:13.914>but <00:06:14.070>then <00:06:14.225>we <00:06:14.380>added <00:06:14.534>a <00:06:14.690>cloud <00:06:14.845>family <00:06:15.000>and <00:06:15.155>Gemini.

00:06:15.310 --> 00:06:15.320 align:start position:0%
Gemini.

00:06:15.320 --> 00:06:17.150 align:start position:0%
<00:06:15.320>Gemini. <00:06:15.553>We <00:06:15.786>also <00:06:16.020>got <00:06:16.253>some <00:06:16.486>interesting <00:06:16.719>feedback

00:06:17.150 --> 00:06:17.160 align:start position:0%
<00:06:17.150>We <00:06:17.151>also <00:06:17.153>got <00:06:17.155>some <00:06:17.157>interesting <00:06:17.159>feedback

00:06:17.160 --> 00:06:18.630 align:start position:0%
<00:06:17.160>We <00:06:17.270>also <00:06:17.380>got <00:06:17.490>some <00:06:17.601>interesting <00:06:17.711>feedback <00:06:17.821>from <00:06:17.931>the <00:06:18.042>other <00:06:18.152>side <00:06:18.262>of <00:06:18.372>the <00:06:18.483>spectrum

00:06:18.630 --> 00:06:18.640 align:start position:0%
<00:06:18.630>from <00:06:18.631>the <00:06:18.633>other <00:06:18.634>side <00:06:18.635>of <00:06:18.637>the <00:06:18.639>spectrum

00:06:18.640 --> 00:06:20.350 align:start position:0%
<00:06:18.640>from <00:06:18.756>the <00:06:18.873>other <00:06:18.990>side <00:06:19.106>of <00:06:19.223>the <00:06:19.340>spectrum <00:06:19.456>where <00:06:19.573>people <00:06:19.690>actually <00:06:19.806>told <00:06:19.923>us, <00:06:20.039>"Hey

00:06:20.350 --> 00:06:20.360 align:start position:0%
<00:06:20.350>where <00:06:20.351>people <00:06:20.353>actually <00:06:20.355>told <00:06:20.357>us, <00:06:20.359>"Hey

00:06:20.360 --> 00:06:22.110 align:start position:0%
<00:06:20.360>where <00:06:20.460>people <00:06:20.560>actually <00:06:20.660>told <00:06:20.760>us, <00:06:20.860>"Hey <00:06:20.960>we're <00:06:21.060>on <00:06:21.160>this <00:06:21.260>and <00:06:21.360>I <00:06:21.460>have <00:06:21.560>no <00:06:21.660>idea <00:06:21.760>what

00:06:22.110 --> 00:06:22.120 align:start position:0%
<00:06:22.110>we're <00:06:22.111>on <00:06:22.112>this <00:06:22.113>and <00:06:22.114>I <00:06:22.115>have <00:06:22.116>no <00:06:22.117>idea <00:06:22.119>what

00:06:22.120 --> 00:06:24.230 align:start position:0%
<00:06:22.120>we're <00:06:22.207>on <00:06:22.295>this <00:06:22.382>and <00:06:22.470>I <00:06:22.557>have <00:06:22.645>no <00:06:22.732>idea <00:06:22.820>what <00:06:22.907>you've <00:06:22.995>done <00:06:23.082>with <00:06:23.170>Ollie <00:06:23.257>but <00:06:23.345>I <00:06:23.432>feel <00:06:23.520>that

00:06:24.230 --> 00:06:24.240 align:start position:0%
<00:06:24.230>you've <00:06:24.231>done <00:06:24.232>with <00:06:24.233>Ollie <00:06:24.235>but <00:06:24.236>I <00:06:24.237>feel <00:06:24.239>that

00:06:24.240 --> 00:06:26.270 align:start position:0%
<00:06:24.240>you've <00:06:24.367>done <00:06:24.494>with <00:06:24.621>Ollie <00:06:24.749>but <00:06:24.876>I <00:06:25.003>feel <00:06:25.130>that <00:06:25.258>now <00:06:25.385>he <00:06:25.512>understands <00:06:25.640>me."

00:06:26.270 --> 00:06:26.280 align:start position:0%
<00:06:26.270>now <00:06:26.272>he <00:06:26.275>understands <00:06:26.279>me."

00:06:26.280 --> 00:06:28.409 align:start position:0%
<00:06:26.280>now <00:06:26.419>he <00:06:26.559>understands <00:06:26.700>me." <00:06:26.840>I <00:06:26.979>was <00:06:27.119>like, <00:06:27.260>"Wow, <00:06:27.400>okay <00:06:27.539>that <00:06:27.679>that

00:06:28.409 --> 00:06:28.419 align:start position:0%
<00:06:28.409>I <00:06:28.410>was <00:06:28.412>like, <00:06:28.413>"Wow, <00:06:28.414>okay <00:06:28.416>that <00:06:28.418>that

00:06:28.419 --> 00:06:28.430 align:start position:0%
<00:06:28.419>I <00:06:28.420>was <00:06:28.421>like, <00:06:28.422>"Wow, <00:06:28.423>okay <00:06:28.425>that <00:06:28.426>that <00:06:28.427>&gt;&gt; <00:06:28.428>[laughter]

00:06:28.430 --> 00:06:28.440 align:start position:0%
<00:06:28.430>&gt;&gt; <00:06:28.439>[laughter]

00:06:28.440 --> 00:06:30.350 align:start position:0%
<00:06:28.440>&gt;&gt; <00:06:28.790>[laughter] <00:06:29.140>&gt;&gt; <00:06:29.490>that's <00:06:29.840>interesting.

00:06:30.350 --> 00:06:30.360 align:start position:0%
<00:06:30.350>&gt;&gt; <00:06:30.354>that's <00:06:30.359>interesting.

00:06:30.360 --> 00:06:31.950 align:start position:0%
<00:06:30.360>&gt;&gt; <00:06:30.500>that's <00:06:30.640>interesting. <00:06:30.780>I <00:06:30.920>don't <00:06:31.060>know <00:06:31.200>what <00:06:31.340>we've <00:06:31.480>done <00:06:31.620>that <00:06:31.760>made

00:06:31.950 --> 00:06:31.960 align:start position:0%
<00:06:31.950>I <00:06:31.951>don't <00:06:31.952>know <00:06:31.953>what <00:06:31.955>we've <00:06:31.956>done <00:06:31.957>that <00:06:31.959>made

00:06:31.960 --> 00:06:34.110 align:start position:0%
<00:06:31.960>I <00:06:32.053>don't <00:06:32.146>know <00:06:32.239>what <00:06:32.333>we've <00:06:32.426>done <00:06:32.520>that <00:06:32.613>made <00:06:32.706>you <00:06:32.799>feel <00:06:32.893>this <00:06:32.986>way <00:06:33.080>but <00:06:33.173>I <00:06:33.266>will <00:06:33.359>check."

00:06:34.110 --> 00:06:34.120 align:start position:0%
<00:06:34.110>you <00:06:34.111>feel <00:06:34.112>this <00:06:34.113>way <00:06:34.115>but <00:06:34.116>I <00:06:34.117>will <00:06:34.119>check."

00:06:34.120 --> 00:06:35.750 align:start position:0%
<00:06:34.120>you <00:06:34.220>feel <00:06:34.320>this <00:06:34.420>way <00:06:34.520>but <00:06:34.620>I <00:06:34.720>will <00:06:34.820>check." <00:06:34.920>And <00:06:35.020>when <00:06:35.120>we <00:06:35.220>later <00:06:35.320>checked <00:06:35.420>the <00:06:35.520>the

00:06:35.750 --> 00:06:35.760 align:start position:0%
<00:06:35.750>And <00:06:35.751>when <00:06:35.753>we <00:06:35.754>later <00:06:35.756>checked <00:06:35.757>the <00:06:35.759>the

00:06:35.760 --> 00:06:38.630 align:start position:0%
<00:06:35.760>And <00:06:35.887>when <00:06:36.014>we <00:06:36.141>later <00:06:36.269>checked <00:06:36.396>the <00:06:36.523>the <00:06:36.650>sessions <00:06:36.778>of <00:06:36.905>this <00:06:37.032>specific <00:06:37.159>customer

00:06:38.630 --> 00:06:38.640 align:start position:0%
<00:06:38.630>sessions <00:06:38.632>of <00:06:38.634>this <00:06:38.636>specific <00:06:38.639>customer

00:06:38.640 --> 00:06:40.030 align:start position:0%
<00:06:38.640>sessions <00:06:38.818>of <00:06:38.997>this <00:06:39.176>specific <00:06:39.354>customer <00:06:39.533>we <00:06:39.712>realized <00:06:39.890>that

00:06:40.030 --> 00:06:40.040 align:start position:0%
<00:06:40.030>we <00:06:40.034>realized <00:06:40.039>that

00:06:40.040 --> 00:06:42.110 align:start position:0%
<00:06:40.040>we <00:06:40.195>realized <00:06:40.351>that <00:06:40.506>all <00:06:40.662>they've <00:06:40.817>done <00:06:40.973>is <00:06:41.128>that <00:06:41.284>they <00:06:41.440>just

00:06:42.110 --> 00:06:42.120 align:start position:0%
<00:06:42.110>all <00:06:42.111>they've <00:06:42.113>done <00:06:42.114>is <00:06:42.116>that <00:06:42.117>they <00:06:42.119>just

00:06:42.120 --> 00:06:44.590 align:start position:0%
<00:06:42.120>all <00:06:42.220>they've <00:06:42.320>done <00:06:42.420>is <00:06:42.520>that <00:06:42.620>they <00:06:42.720>just <00:06:42.820>changed <00:06:42.920>the <00:06:43.020>model <00:06:43.120>from <00:06:43.220>GPT <00:06:43.320>to <00:06:43.420>Claude <00:06:43.520>and

00:06:44.590 --> 00:06:44.600 align:start position:0%
<00:06:44.590>changed <00:06:44.591>the <00:06:44.592>model <00:06:44.593>from <00:06:44.595>GPT <00:06:44.596>to <00:06:44.597>Claude <00:06:44.599>and

00:06:44.600 --> 00:06:45.950 align:start position:0%
<00:06:44.600>changed <00:06:44.710>the <00:06:44.820>model <00:06:44.931>from <00:06:45.041>GPT <00:06:45.152>to <00:06:45.262>Claude <00:06:45.373>and <00:06:45.483>that <00:06:45.594>completely <00:06:45.704>changed <00:06:45.815>their

00:06:45.950 --> 00:06:45.960 align:start position:0%
<00:06:45.950>that <00:06:45.953>completely <00:06:45.955>changed <00:06:45.959>their

00:06:45.960 --> 00:06:47.190 align:start position:0%
<00:06:45.960>that <00:06:46.236>completely <00:06:46.513>changed <00:06:46.790>their <00:06:47.067>experience.

00:06:47.190 --> 00:06:47.200 align:start position:0%
experience.

00:06:47.200 --> 00:06:49.590 align:start position:0%
<00:06:47.200>experience. <00:06:47.433>Same <00:06:47.666>system <00:06:47.900>prompt, <00:06:48.133>same <00:06:48.366>tools, <00:06:48.599>same

00:06:49.590 --> 00:06:49.600 align:start position:0%
<00:06:49.590>Same <00:06:49.591>system <00:06:49.593>prompt, <00:06:49.595>same <00:06:49.597>tools, <00:06:49.599>same

00:06:49.600 --> 00:06:50.790 align:start position:0%
<00:06:49.600>Same <00:06:49.778>system <00:06:49.957>prompt, <00:06:50.135>same <00:06:50.314>tools, <00:06:50.492>same <00:06:50.671>agent.

00:06:50.790 --> 00:06:50.800 align:start position:0%
agent.

00:06:50.800 --> 00:06:52.310 align:start position:0%
<00:06:50.800>agent. <00:06:51.071>Different <00:06:51.343>model <00:06:51.615>got <00:06:51.887>different <00:06:52.159>behavior.

00:06:52.310 --> 00:06:52.320 align:start position:0%
<00:06:52.310>Different <00:06:52.312>model <00:06:52.314>got <00:06:52.316>different <00:06:52.319>behavior.

00:06:52.320 --> 00:06:54.470 align:start position:0%
<00:06:52.320>Different <00:06:52.427>model <00:06:52.535>got <00:06:52.643>different <00:06:52.750>behavior. <00:06:52.858>So <00:06:52.966>it <00:06:53.073>could <00:06:53.181>be <00:06:53.289>bad, <00:06:53.396>it <00:06:53.504>could <00:06:53.612>be <00:06:53.719>good.

00:06:54.470 --> 00:06:54.480 align:start position:0%
<00:06:54.470>So <00:06:54.471>it <00:06:54.472>could <00:06:54.473>be <00:06:54.474>bad, <00:06:54.475>it <00:06:54.476>could <00:06:54.477>be <00:06:54.479>good.

00:06:54.480 --> 00:06:56.830 align:start position:0%
<00:06:54.480>So <00:06:54.573>it <00:06:54.666>could <00:06:54.760>be <00:06:54.853>bad, <00:06:54.946>it <00:06:55.040>could <00:06:55.133>be <00:06:55.226>good. <00:06:55.320>But <00:06:55.413>that's <00:06:55.506>where <00:06:55.600>we <00:06:55.693>as <00:06:55.786>the <00:06:55.880>team

00:06:56.830 --> 00:06:56.840 align:start position:0%
<00:06:56.830>But <00:06:56.831>that's <00:06:56.832>where <00:06:56.834>we <00:06:56.836>as <00:06:56.837>the <00:06:56.839>team

00:06:56.840 --> 00:06:59.270 align:start position:0%
<00:06:56.840>But <00:06:56.995>that's <00:06:57.151>where <00:06:57.306>we <00:06:57.462>as <00:06:57.617>the <00:06:57.773>team <00:06:57.928>developing, <00:06:58.084>maintaining, <00:06:58.240>monitoring

00:06:59.270 --> 00:06:59.280 align:start position:0%
<00:06:59.270>developing, <00:06:59.274>maintaining, <00:06:59.279>monitoring

00:06:59.280 --> 00:07:00.870 align:start position:0%
<00:06:59.280>developing, <00:06:59.455>maintaining, <00:06:59.630>monitoring <00:06:59.804>these <00:06:59.979>agents, <00:07:00.155>this <00:07:00.330>is <00:07:00.505>where <00:07:00.679>our

00:07:00.870 --> 00:07:00.880 align:start position:0%
<00:07:00.870>these <00:07:00.871>agents, <00:07:00.873>this <00:07:00.875>is <00:07:00.877>where <00:07:00.879>our

00:07:00.880 --> 00:07:03.430 align:start position:0%
<00:07:00.880>these <00:07:01.035>agents, <00:07:01.191>this <00:07:01.346>is <00:07:01.502>where <00:07:01.657>our <00:07:01.813>responsibility <00:07:01.968>actually <00:07:02.124>to <00:07:02.280>evaluate.

00:07:03.430 --> 00:07:03.440 align:start position:0%
<00:07:03.430>responsibility <00:07:03.433>actually <00:07:03.436>to <00:07:03.439>evaluate.

00:07:03.440 --> 00:07:05.190 align:start position:0%
<00:07:03.440>responsibility <00:07:03.719>actually <00:07:04.000>to <00:07:04.280>evaluate. <00:07:04.560>Um <00:07:04.840>so

00:07:05.190 --> 00:07:05.200 align:start position:0%
<00:07:05.190>Um <00:07:05.199>so

00:07:05.200 --> 00:07:07.510 align:start position:0%
<00:07:05.200>Um <00:07:05.375>so <00:07:05.550>the <00:07:05.724>entire <00:07:05.900>process <00:07:06.075>we <00:07:06.250>go <00:07:06.425>through <00:07:06.599>today

00:07:07.510 --> 00:07:07.520 align:start position:0%
<00:07:07.510>the <00:07:07.511>entire <00:07:07.513>process <00:07:07.514>we <00:07:07.515>go <00:07:07.517>through <00:07:07.519>today

00:07:07.520 --> 00:07:09.630 align:start position:0%
<00:07:07.520>the <00:07:07.659>entire <00:07:07.799>process <00:07:07.940>we <00:07:08.080>go <00:07:08.219>through <00:07:08.359>today <00:07:08.500>with <00:07:08.640>every <00:07:08.780>new <00:07:08.919>model.

00:07:09.630 --> 00:07:09.640 align:start position:0%
<00:07:09.630>with <00:07:09.633>every <00:07:09.635>new <00:07:09.639>model.

00:07:09.640 --> 00:07:13.070 align:start position:0%
<00:07:09.640>with <00:07:09.756>every <00:07:09.873>new <00:07:09.990>model. <00:07:10.106>Right? <00:07:10.223>We <00:07:10.340>just <00:07:10.456>got <00:07:10.573>now <00:07:10.690>Sonnet <00:07:10.806>5.0. <00:07:10.923>We <00:07:11.039>we

00:07:13.070 --> 00:07:13.080 align:start position:0%
<00:07:13.070>Right? <00:07:13.071>We <00:07:13.072>just <00:07:13.073>got <00:07:13.074>now <00:07:13.075>Sonnet <00:07:13.076>5.0. <00:07:13.077>We <00:07:13.079>we

00:07:13.080 --> 00:07:15.510 align:start position:0%
<00:07:13.080>Right? <00:07:13.187>We <00:07:13.295>just <00:07:13.403>got <00:07:13.510>now <00:07:13.618>Sonnet <00:07:13.726>5.0. <00:07:13.833>We <00:07:13.941>we <00:07:14.049>got <00:07:14.156>new <00:07:14.264>models <00:07:14.372>from <00:07:14.479>GPT.

00:07:15.510 --> 00:07:15.520 align:start position:0%
<00:07:15.510>got <00:07:15.512>new <00:07:15.514>models <00:07:15.516>from <00:07:15.519>GPT.

00:07:15.520 --> 00:07:17.430 align:start position:0%
<00:07:15.520>got <00:07:15.636>new <00:07:15.753>models <00:07:15.870>from <00:07:15.986>GPT. <00:07:16.103>I <00:07:16.219>can <00:07:16.336>tell <00:07:16.453>you <00:07:16.570>we <00:07:16.686>haven't <00:07:16.803>yet <00:07:16.919>released

00:07:17.430 --> 00:07:17.440 align:start position:0%
<00:07:17.430>I <00:07:17.431>can <00:07:17.432>tell <00:07:17.433>you <00:07:17.435>we <00:07:17.436>haven't <00:07:17.437>yet <00:07:17.439>released

00:07:17.440 --> 00:07:19.430 align:start position:0%
<00:07:17.440>I <00:07:17.556>can <00:07:17.673>tell <00:07:17.790>you <00:07:17.906>we <00:07:18.023>haven't <00:07:18.140>yet <00:07:18.256>released <00:07:18.373>it <00:07:18.490>to <00:07:18.606>our <00:07:18.723>customers. <00:07:18.840>Why?

00:07:19.430 --> 00:07:19.440 align:start position:0%
<00:07:19.430>it <00:07:19.432>to <00:07:19.434>our <00:07:19.436>customers. <00:07:19.439>Why?

00:07:19.440 --> 00:07:20.950 align:start position:0%
<00:07:19.440>it <00:07:19.553>to <00:07:19.666>our <00:07:19.779>customers. <00:07:19.893>Why? <00:07:20.006>Because <00:07:20.119>it <00:07:20.232>takes <00:07:20.346>us <00:07:20.459>time. <00:07:20.572>It's <00:07:20.685>not <00:07:20.799>just

00:07:20.950 --> 00:07:20.960 align:start position:0%
<00:07:20.950>Because <00:07:20.951>it <00:07:20.952>takes <00:07:20.953>us <00:07:20.955>time. <00:07:20.956>It's <00:07:20.957>not <00:07:20.959>just

00:07:20.960 --> 00:07:23.510 align:start position:0%
<00:07:20.960>Because <00:07:21.053>it <00:07:21.146>takes <00:07:21.239>us <00:07:21.333>time. <00:07:21.426>It's <00:07:21.520>not <00:07:21.613>just <00:07:21.706>a <00:07:21.799>one-day <00:07:21.893>task. <00:07:21.986>On <00:07:22.080>average <00:07:22.173>it's <00:07:22.266>about <00:07:22.359>2

00:07:23.510 --> 00:07:23.520 align:start position:0%
<00:07:23.510>a <00:07:23.511>one-day <00:07:23.512>task. <00:07:23.513>On <00:07:23.515>average <00:07:23.516>it's <00:07:23.517>about <00:07:23.519>2

00:07:23.520 --> 00:07:25.790 align:start position:0%
<00:07:23.520>a <00:07:23.620>one-day <00:07:23.719>task. <00:07:23.820>On <00:07:23.919>average <00:07:24.020>it's <00:07:24.120>about <00:07:24.219>2 <00:07:24.320>weeks <00:07:24.419>just <00:07:24.520>to <00:07:24.620>evaluate <00:07:24.719>these <00:07:24.820>things <00:07:24.919>and

00:07:25.790 --> 00:07:25.800 align:start position:0%
<00:07:25.790>weeks <00:07:25.791>just <00:07:25.793>to <00:07:25.794>evaluate <00:07:25.796>these <00:07:25.797>things <00:07:25.799>and

00:07:25.800 --> 00:07:28.150 align:start position:0%
<00:07:25.800>weeks <00:07:25.900>just <00:07:26.000>to <00:07:26.100>evaluate <00:07:26.200>these <00:07:26.300>things <00:07:26.400>and <00:07:26.500>to <00:07:26.600>make <00:07:26.700>sure <00:07:26.800>one <00:07:26.900>we <00:07:27.000>run <00:07:27.100>evaluations, <00:07:27.200>we

00:07:28.150 --> 00:07:28.160 align:start position:0%
<00:07:28.150>to <00:07:28.151>make <00:07:28.152>sure <00:07:28.153>one <00:07:28.155>we <00:07:28.156>run <00:07:28.157>evaluations, <00:07:28.159>we

00:07:28.160 --> 00:07:30.270 align:start position:0%
<00:07:28.160>to <00:07:28.276>make <00:07:28.393>sure <00:07:28.510>one <00:07:28.626>we <00:07:28.743>run <00:07:28.860>evaluations, <00:07:28.976>we <00:07:29.093>then <00:07:29.210>fine-tune <00:07:29.326>system <00:07:29.443>prompt, <00:07:29.560>tool

00:07:30.270 --> 00:07:30.280 align:start position:0%
<00:07:30.270>then <00:07:30.272>fine-tune <00:07:30.274>system <00:07:30.276>prompt, <00:07:30.279>tool

00:07:30.280 --> 00:07:33.110 align:start position:0%
<00:07:30.280>then <00:07:30.407>fine-tune <00:07:30.534>system <00:07:30.661>prompt, <00:07:30.789>tool <00:07:30.916>calls, <00:07:31.043>our <00:07:31.170>entire <00:07:31.298>agent <00:07:31.425>actually <00:07:31.552>to <00:07:31.679>make

00:07:33.110 --> 00:07:33.120 align:start position:0%
<00:07:33.110>calls, <00:07:33.111>our <00:07:33.113>entire <00:07:33.114>agent <00:07:33.116>actually <00:07:33.117>to <00:07:33.119>make

00:07:33.120 --> 00:07:35.630 align:start position:0%
<00:07:33.120>calls, <00:07:33.247>our <00:07:33.374>entire <00:07:33.501>agent <00:07:33.629>actually <00:07:33.756>to <00:07:33.883>make <00:07:34.010>sure <00:07:34.138>it <00:07:34.265>does <00:07:34.392>gain <00:07:34.520>improvement.

00:07:35.630 --> 00:07:35.640 align:start position:0%
<00:07:35.630>sure <00:07:35.632>it <00:07:35.634>does <00:07:35.636>gain <00:07:35.639>improvement.

00:07:35.640 --> 00:07:38.470 align:start position:0%
<00:07:35.640>sure <00:07:35.780>it <00:07:35.919>does <00:07:36.060>gain <00:07:36.200>improvement. <00:07:36.340>Um <00:07:36.479>then <00:07:36.620>we <00:07:36.760>release <00:07:36.900>in <00:07:37.039>canary.

00:07:38.470 --> 00:07:38.480 align:start position:0%
<00:07:38.470>Um <00:07:38.471>then <00:07:38.473>we <00:07:38.475>release <00:07:38.477>in <00:07:38.479>canary.

00:07:38.480 --> 00:07:41.350 align:start position:0%
<00:07:38.480>Um <00:07:38.573>then <00:07:38.666>we <00:07:38.760>release <00:07:38.853>in <00:07:38.946>canary. <00:07:39.040>So <00:07:39.133>we <00:07:39.226>want <00:07:39.320>to <00:07:39.413>see <00:07:39.506>how <00:07:39.600>are <00:07:39.693>you <00:07:39.786>the <00:07:39.880>users'

00:07:41.350 --> 00:07:41.360 align:start position:0%
<00:07:41.350>So <00:07:41.351>we <00:07:41.352>want <00:07:41.353>to <00:07:41.354>see <00:07:41.355>how <00:07:41.356>are <00:07:41.357>you <00:07:41.358>the <00:07:41.359>users'

00:07:41.360 --> 00:07:43.070 align:start position:0%
<00:07:41.360>So <00:07:41.460>we <00:07:41.560>want <00:07:41.660>to <00:07:41.760>see <00:07:41.860>how <00:07:41.960>are <00:07:42.060>you <00:07:42.160>the <00:07:42.260>users' <00:07:42.360>experience <00:07:42.460>actually <00:07:42.560>is <00:07:42.660>being <00:07:42.760>affected

00:07:43.070 --> 00:07:43.080 align:start position:0%
<00:07:43.070>experience <00:07:43.072>actually <00:07:43.074>is <00:07:43.076>being <00:07:43.079>affected

00:07:43.080 --> 00:07:46.390 align:start position:0%
<00:07:43.080>experience <00:07:43.196>actually <00:07:43.313>is <00:07:43.430>being <00:07:43.546>affected <00:07:43.663>for <00:07:43.780>good <00:07:43.896>for <00:07:44.013>bad. <00:07:44.130>Um <00:07:44.246>and <00:07:44.363>we <00:07:44.479>constantly

00:07:46.390 --> 00:07:46.400 align:start position:0%
<00:07:46.390>for <00:07:46.391>good <00:07:46.392>for <00:07:46.393>bad. <00:07:46.395>Um <00:07:46.396>and <00:07:46.397>we <00:07:46.399>constantly

00:07:46.400 --> 00:07:48.750 align:start position:0%
<00:07:46.400>for <00:07:46.493>good <00:07:46.586>for <00:07:46.679>bad. <00:07:46.773>Um <00:07:46.866>and <00:07:46.960>we <00:07:47.053>constantly <00:07:47.146>monitor <00:07:47.239>the <00:07:47.333>behavior <00:07:47.426>of <00:07:47.520>our <00:07:47.613>users <00:07:47.706>of <00:07:47.799>the

00:07:48.750 --> 00:07:48.760 align:start position:0%
<00:07:48.750>monitor <00:07:48.751>the <00:07:48.752>behavior <00:07:48.753>of <00:07:48.755>our <00:07:48.756>users <00:07:48.757>of <00:07:48.759>the

00:07:48.760 --> 00:07:51.590 align:start position:0%
<00:07:48.760>monitor <00:07:48.876>the <00:07:48.993>behavior <00:07:49.110>of <00:07:49.226>our <00:07:49.343>users <00:07:49.460>of <00:07:49.576>the <00:07:49.693>experience <00:07:49.810>they <00:07:49.926>get <00:07:50.043>with <00:07:50.159>Ollie.

00:07:51.590 --> 00:07:51.600 align:start position:0%
<00:07:51.590>experience <00:07:51.592>they <00:07:51.594>get <00:07:51.596>with <00:07:51.599>Ollie.

00:07:51.600 --> 00:07:55.670 align:start position:0%
<00:07:51.600>experience <00:07:51.716>they <00:07:51.833>get <00:07:51.950>with <00:07:52.066>Ollie. <00:07:52.183>So, <00:07:52.300>um <00:07:52.416>that <00:07:52.533>was <00:07:52.650>August <00:07:52.766>2025. <00:07:52.883>We <00:07:53.000>learned

00:07:55.670 --> 00:07:55.680 align:start position:0%
<00:07:55.670>So, <00:07:55.671>um <00:07:55.672>that <00:07:55.673>was <00:07:55.675>August <00:07:55.676>2025. <00:07:55.677>We <00:07:55.679>learned

00:07:55.680 --> 00:07:58.030 align:start position:0%
<00:07:55.680>So, <00:07:55.780>um <00:07:55.880>that <00:07:55.980>was <00:07:56.080>August <00:07:56.180>2025. <00:07:56.280>We <00:07:56.380>learned <00:07:56.480>we <00:07:56.580>we <00:07:56.680>now <00:07:56.780>do <00:07:56.880>evaluations. <00:07:56.980>We <00:07:57.080>don't

00:07:58.030 --> 00:07:58.040 align:start position:0%
<00:07:58.030>we <00:07:58.031>we <00:07:58.033>now <00:07:58.034>do <00:07:58.036>evaluations. <00:07:58.037>We <00:07:58.039>don't

00:07:58.040 --> 00:08:01.390 align:start position:0%
<00:07:58.040>we <00:07:58.156>we <00:07:58.273>now <00:07:58.390>do <00:07:58.506>evaluations. <00:07:58.623>We <00:07:58.740>don't <00:07:58.856>change <00:07:58.973>models <00:07:59.090>that <00:07:59.206>often. <00:07:59.323>Um <00:07:59.440>December

00:08:01.390 --> 00:08:01.400 align:start position:0%
<00:08:01.390>change <00:08:01.391>models <00:08:01.393>that <00:08:01.395>often. <00:08:01.397>Um <00:08:01.399>December

00:08:01.400 --> 00:08:03.510 align:start position:0%
<00:08:01.400>change <00:08:01.633>models <00:08:01.866>that <00:08:02.099>often. <00:08:02.333>Um <00:08:02.566>December <00:08:02.799>2025,

00:08:03.510 --> 00:08:03.520 align:start position:0%
2025,

00:08:03.520 --> 00:08:06.270 align:start position:0%
<00:08:03.520>2025, <00:08:03.719>we <00:08:03.919>launched <00:08:04.120>the <00:08:04.320>AI. <00:08:04.520>It <00:08:04.719>was <00:08:04.919>amazing.

00:08:06.270 --> 00:08:06.280 align:start position:0%
<00:08:06.270>we <00:08:06.271>launched <00:08:06.272>the <00:08:06.274>AI. <00:08:06.275>It <00:08:06.277>was <00:08:06.279>amazing.

00:08:06.280 --> 00:08:08.110 align:start position:0%
<00:08:06.280>we <00:08:06.373>launched <00:08:06.466>the <00:08:06.559>AI. <00:08:06.653>It <00:08:06.746>was <00:08:06.840>amazing. <00:08:06.933>We <00:08:07.026>opened <00:08:07.119>the <00:08:07.213>the <00:08:07.306>doors <00:08:07.400>to <00:08:07.493>all <00:08:07.586>of <00:08:07.679>our

00:08:08.110 --> 00:08:08.120 align:start position:0%
<00:08:08.110>We <00:08:08.111>opened <00:08:08.112>the <00:08:08.113>the <00:08:08.114>doors <00:08:08.115>to <00:08:08.116>all <00:08:08.117>of <00:08:08.119>our

00:08:08.120 --> 00:08:09.550 align:start position:0%
<00:08:08.120>We <00:08:08.263>opened <00:08:08.406>the <00:08:08.549>the <00:08:08.692>doors <00:08:08.835>to <00:08:08.978>all <00:08:09.121>of <00:08:09.264>our <00:08:09.407>customers.

00:08:09.550 --> 00:08:09.560 align:start position:0%
customers.

00:08:09.560 --> 00:08:11.030 align:start position:0%
<00:08:09.560>customers. <00:08:09.780>We <00:08:10.001>started <00:08:10.221>to <00:08:10.442>get <00:08:10.662>some <00:08:10.883>amazing

00:08:11.030 --> 00:08:11.040 align:start position:0%
<00:08:11.030>We <00:08:11.031>started <00:08:11.033>to <00:08:11.035>get <00:08:11.037>some <00:08:11.039>amazing

00:08:11.040 --> 00:08:13.350 align:start position:0%
<00:08:11.040>We <00:08:11.156>started <00:08:11.273>to <00:08:11.390>get <00:08:11.506>some <00:08:11.623>amazing <00:08:11.740>feedbacks. <00:08:11.856>Um <00:08:11.973>people <00:08:12.090>were <00:08:12.206>amazed <00:08:12.323>by <00:08:12.440>the

00:08:13.350 --> 00:08:13.360 align:start position:0%
<00:08:13.350>feedbacks. <00:08:13.351>Um <00:08:13.353>people <00:08:13.354>were <00:08:13.356>amazed <00:08:13.357>by <00:08:13.359>the

00:08:13.360 --> 00:08:16.030 align:start position:0%
<00:08:13.360>feedbacks. <00:08:13.467>Um <00:08:13.575>people <00:08:13.683>were <00:08:13.790>amazed <00:08:13.898>by <00:08:14.006>the <00:08:14.113>capabilities <00:08:14.221>they <00:08:14.329>that <00:08:14.436>were <00:08:14.544>shown <00:08:14.652>in <00:08:14.760>in

00:08:16.030 --> 00:08:16.040 align:start position:0%
<00:08:16.030>capabilities <00:08:16.031>they <00:08:16.033>that <00:08:16.034>were <00:08:16.036>shown <00:08:16.037>in <00:08:16.039>in

00:08:16.040 --> 00:08:19.750 align:start position:0%
<00:08:16.040>capabilities <00:08:16.180>they <00:08:16.320>that <00:08:16.460>were <00:08:16.600>shown <00:08:16.740>in <00:08:16.880>in <00:08:17.020>actually <00:08:17.160>available <00:08:17.300>through <00:08:17.440>Ollie.

00:08:19.760 --> 00:08:21.110 align:start position:0%
<00:08:19.760>And <00:08:20.975>then

00:08:21.110 --> 00:08:21.120 align:start position:0%
<00:08:21.110>And <00:08:21.119>then

00:08:21.120 --> 00:08:23.950 align:start position:0%
<00:08:21.120>And <00:08:21.295>then <00:08:21.470>something <00:08:21.645>else <00:08:21.820>else <00:08:21.995>came <00:08:22.170>up. <00:08:22.345>Um <00:08:22.520>users

00:08:23.950 --> 00:08:23.960 align:start position:0%
<00:08:23.950>something <00:08:23.951>else <00:08:23.953>else <00:08:23.954>came <00:08:23.955>up. <00:08:23.957>Um <00:08:23.959>users

00:08:23.960 --> 00:08:26.390 align:start position:0%
<00:08:23.960>something <00:08:24.135>else <00:08:24.310>else <00:08:24.484>came <00:08:24.659>up. <00:08:24.835>Um <00:08:25.010>users <00:08:25.185>starts <00:08:25.359>complaining

00:08:26.390 --> 00:08:26.400 align:start position:0%
<00:08:26.390>starts <00:08:26.399>complaining

00:08:26.400 --> 00:08:29.710 align:start position:0%
<00:08:26.400>starts <00:08:26.555>complaining <00:08:26.711>that <00:08:26.866>Ollie <00:08:27.022>is <00:08:27.177>consuming <00:08:27.333>a <00:08:27.488>lot <00:08:27.644>of <00:08:27.799>tokens.

00:08:29.710 --> 00:08:29.720 align:start position:0%
<00:08:29.710>that <00:08:29.711>Ollie <00:08:29.712>is <00:08:29.713>consuming <00:08:29.715>a <00:08:29.716>lot <00:08:29.717>of <00:08:29.719>tokens.

00:08:29.720 --> 00:08:31.430 align:start position:0%
<00:08:29.720>that <00:08:29.807>Ollie <00:08:29.895>is <00:08:29.982>consuming <00:08:30.070>a <00:08:30.157>lot <00:08:30.245>of <00:08:30.332>tokens. <00:08:30.420>Now, <00:08:30.507>when <00:08:30.595>I <00:08:30.682>heard <00:08:30.770>this <00:08:30.857>the <00:08:30.945>first <00:08:31.032>time, <00:08:31.120>I

00:08:31.430 --> 00:08:31.440 align:start position:0%
<00:08:31.430>Now, <00:08:31.431>when <00:08:31.432>I <00:08:31.433>heard <00:08:31.434>this <00:08:31.435>the <00:08:31.436>first <00:08:31.437>time, <00:08:31.439>I

00:08:31.440 --> 00:08:33.350 align:start position:0%
<00:08:31.440>Now, <00:08:31.533>when <00:08:31.626>I <00:08:31.719>heard <00:08:31.813>this <00:08:31.906>the <00:08:32.000>first <00:08:32.093>time, <00:08:32.186>I <00:08:32.280>was <00:08:32.373>skeptical. <00:08:32.466>I <00:08:32.559>was <00:08:32.653>like, <00:08:32.746>"What <00:08:32.840>are

00:08:33.350 --> 00:08:33.360 align:start position:0%
<00:08:33.350>was <00:08:33.351>skeptical. <00:08:33.353>I <00:08:33.354>was <00:08:33.356>like, <00:08:33.357>"What <00:08:33.359>are

00:08:33.360 --> 00:08:35.589 align:start position:0%
<00:08:33.360>was <00:08:33.460>skeptical. <00:08:33.560>I <00:08:33.659>was <00:08:33.760>like, <00:08:33.860>"What <00:08:33.960>are <00:08:34.060>they <00:08:34.159>crying <00:08:34.260>about? <00:08:34.360>You <00:08:34.460>know, <00:08:34.560>like <00:08:34.659>how <00:08:34.760>do

00:08:35.589 --> 00:08:35.599 align:start position:0%
<00:08:35.588>they <00:08:35.590>crying <00:08:35.591>about? <00:08:35.592>You <00:08:35.594>know, <00:08:35.595>like <00:08:35.596>how <00:08:35.598>do

00:08:35.599 --> 00:08:37.430 align:start position:0%
<00:08:35.599>they <00:08:35.692>crying <00:08:35.785>about? <00:08:35.879>You <00:08:35.972>know, <00:08:36.065>like <00:08:36.159>how <00:08:36.252>do <00:08:36.345>they <00:08:36.439>have <00:08:36.532>any <00:08:36.625>idea <00:08:36.719>if <00:08:36.812>this <00:08:36.905>is <00:08:36.999>many

00:08:37.430 --> 00:08:37.440 align:start position:0%
<00:08:37.429>they <00:08:37.431>have <00:08:37.432>any <00:08:37.433>idea <00:08:37.435>if <00:08:37.436>this <00:08:37.437>is <00:08:37.439>many

00:08:37.440 --> 00:08:39.469 align:start position:0%
<00:08:37.440>they <00:08:37.540>have <00:08:37.640>any <00:08:37.740>idea <00:08:37.840>if <00:08:37.940>this <00:08:38.040>is <00:08:38.140>many <00:08:38.240>tokens <00:08:38.340>or <00:08:38.440>this <00:08:38.540>is <00:08:38.640>small <00:08:38.740>amount <00:08:38.840>of

00:08:39.469 --> 00:08:39.479 align:start position:0%
<00:08:39.469>tokens <00:08:39.470>or <00:08:39.472>this <00:08:39.473>is <00:08:39.475>small <00:08:39.476>amount <00:08:39.478>of

00:08:39.479 --> 00:08:41.630 align:start position:0%
<00:08:39.479>tokens <00:08:39.586>or <00:08:39.694>this <00:08:39.802>is <00:08:39.909>small <00:08:40.017>amount <00:08:40.125>of <00:08:40.232>tokens? <00:08:40.340>It's <00:08:40.448>our <00:08:40.555>agent <00:08:40.663>that <00:08:40.771>they <00:08:40.879>they

00:08:41.630 --> 00:08:41.640 align:start position:0%
<00:08:41.630>tokens? <00:08:41.631>It's <00:08:41.633>our <00:08:41.634>agent <00:08:41.635>that <00:08:41.637>they <00:08:41.639>they

00:08:41.640 --> 00:08:44.790 align:start position:0%
<00:08:41.640>tokens? <00:08:41.747>It's <00:08:41.855>our <00:08:41.963>agent <00:08:42.070>that <00:08:42.178>they <00:08:42.286>they <00:08:42.393>don't <00:08:42.501>actually <00:08:42.609>how <00:08:42.716>it <00:08:42.824>works." <00:08:42.932>Um <00:08:43.039>and

00:08:44.790 --> 00:08:44.800 align:start position:0%
<00:08:44.790>don't <00:08:44.791>actually <00:08:44.793>how <00:08:44.794>it <00:08:44.795>works." <00:08:44.797>Um <00:08:44.799>and

00:08:44.800 --> 00:08:46.510 align:start position:0%
<00:08:44.800>don't <00:08:44.927>actually <00:08:45.054>how <00:08:45.181>it <00:08:45.309>works." <00:08:45.436>Um <00:08:45.563>and <00:08:45.690>honestly <00:08:45.818>behind <00:08:45.945>the <00:08:46.072>scenes <00:08:46.199>processing

00:08:46.510 --> 00:08:46.520 align:start position:0%
<00:08:46.510>honestly <00:08:46.512>behind <00:08:46.514>the <00:08:46.516>scenes <00:08:46.519>processing

00:08:46.520 --> 00:08:47.870 align:start position:0%
<00:08:46.520>honestly <00:08:46.655>behind <00:08:46.790>the <00:08:46.925>scenes <00:08:47.060>processing <00:08:47.194>logs <00:08:47.330>and <00:08:47.465>metrics <00:08:47.600>and <00:08:47.735>production

00:08:47.870 --> 00:08:47.880 align:start position:0%
<00:08:47.870>logs <00:08:47.872>and <00:08:47.874>metrics <00:08:47.876>and <00:08:47.879>production

00:08:47.880 --> 00:08:49.230 align:start position:0%
<00:08:47.880>logs <00:08:48.123>and <00:08:48.366>metrics <00:08:48.609>and <00:08:48.852>production <00:08:49.095>telemetry

00:08:49.230 --> 00:08:49.240 align:start position:0%
telemetry

00:08:49.240 --> 00:08:51.430 align:start position:0%
<00:08:49.240>telemetry <00:08:49.395>does <00:08:49.551>come <00:08:49.706>with <00:08:49.862>a <00:08:50.017>lot <00:08:50.173>of <00:08:50.328>data, <00:08:50.484>right? <00:08:50.640>So,

00:08:51.430 --> 00:08:51.440 align:start position:0%
<00:08:51.430>does <00:08:51.431>come <00:08:51.432>with <00:08:51.433>a <00:08:51.434>lot <00:08:51.435>of <00:08:51.436>data, <00:08:51.437>right? <00:08:51.439>So,

00:08:51.440 --> 00:08:53.230 align:start position:0%
<00:08:51.440>does <00:08:51.522>come <00:08:51.604>with <00:08:51.687>a <00:08:51.769>lot <00:08:51.851>of <00:08:51.934>data, <00:08:52.016>right? <00:08:52.098>So, <00:08:52.181>it <00:08:52.263>was <00:08:52.345>it <00:08:52.428>didn't <00:08:52.510>come <00:08:52.592>to <00:08:52.675>us <00:08:52.757>as <00:08:52.840>a

00:08:53.230 --> 00:08:53.240 align:start position:0%
<00:08:53.230>it <00:08:53.231>was <00:08:53.232>it <00:08:53.233>didn't <00:08:53.234>come <00:08:53.235>to <00:08:53.236>us <00:08:53.237>as <00:08:53.239>a

00:08:53.240 --> 00:08:56.230 align:start position:0%
<00:08:53.240>it <00:08:53.327>was <00:08:53.415>it <00:08:53.502>didn't <00:08:53.590>come <00:08:53.677>to <00:08:53.765>us <00:08:53.852>as <00:08:53.940>a <00:08:54.027>surprise <00:08:54.115>that <00:08:54.202>it <00:08:54.290>could <00:08:54.377>have <00:08:54.465>it <00:08:54.552>could <00:08:54.640>be

00:08:56.230 --> 00:08:56.240 align:start position:0%
<00:08:56.230>surprise <00:08:56.231>that <00:08:56.232>it <00:08:56.233>could <00:08:56.235>have <00:08:56.236>it <00:08:56.237>could <00:08:56.239>be

00:08:56.240 --> 00:08:57.990 align:start position:0%
<00:08:56.240>surprise <00:08:56.340>that <00:08:56.440>it <00:08:56.540>could <00:08:56.640>have <00:08:56.740>it <00:08:56.840>could <00:08:56.940>be <00:08:57.040>that <00:08:57.140>this <00:08:57.240>is <00:08:57.340>a <00:08:57.440>lot <00:08:57.540>of <00:08:57.640>um

00:08:57.990 --> 00:08:58.000 align:start position:0%
<00:08:57.990>that <00:08:57.991>this <00:08:57.993>is <00:08:57.994>a <00:08:57.996>lot <00:08:57.997>of <00:08:57.999>um

00:08:58.000 --> 00:09:00.371 align:start position:0%
<00:08:58.000>that <00:08:58.116>this <00:08:58.233>is <00:08:58.350>a <00:08:58.466>lot <00:08:58.583>of <00:08:58.700>um <00:08:58.816>data <00:08:58.933>or <00:08:59.050>a <00:08:59.166>lot <00:08:59.283>of <00:08:59.400>tokens.

00:09:00.371 --> 00:09:00.381 align:start position:0%
<00:09:00.371>data <00:09:00.372>or <00:09:00.374>a <00:09:00.376>lot <00:09:00.378>of <00:09:00.380>tokens.

00:09:00.381 --> 00:09:01.390 align:start position:0%
<00:09:00.381>data <00:09:00.510>or <00:09:00.640>a <00:09:00.770>lot <00:09:00.899>of <00:09:01.029>tokens. <00:09:01.159>&gt;&gt; <00:09:01.289>[snorts]

00:09:01.390 --> 00:09:01.400 align:start position:0%
<00:09:01.390>&gt;&gt; <00:09:01.399>[snorts]

00:09:01.400 --> 00:09:04.070 align:start position:0%
<00:09:01.400>&gt;&gt; <00:09:01.555>[snorts] <00:09:01.711>&gt;&gt; <00:09:01.866>But <00:09:02.022>um <00:09:02.177>one <00:09:02.333>of <00:09:02.488>our <00:09:02.644>research <00:09:02.800>engineers

00:09:04.070 --> 00:09:04.080 align:start position:0%
<00:09:04.070>&gt;&gt; <00:09:04.071>But <00:09:04.072>um <00:09:04.073>one <00:09:04.075>of <00:09:04.076>our <00:09:04.077>research <00:09:04.079>engineers

00:09:04.080 --> 00:09:07.510 align:start position:0%
<00:09:04.080>&gt;&gt; <00:09:04.187>But <00:09:04.295>um <00:09:04.403>one <00:09:04.510>of <00:09:04.618>our <00:09:04.726>research <00:09:04.833>engineers <00:09:04.941>actually <00:09:05.049>um <00:09:05.156>decided <00:09:05.264>to <00:09:05.372>investigate <00:09:05.480>this

00:09:07.510 --> 00:09:07.520 align:start position:0%
<00:09:07.510>actually <00:09:07.511>um <00:09:07.513>decided <00:09:07.515>to <00:09:07.517>investigate <00:09:07.519>this

00:09:07.520 --> 00:09:08.590 align:start position:0%
<00:09:07.520>actually <00:09:07.680>um <00:09:07.841>decided <00:09:08.001>to <00:09:08.162>investigate <00:09:08.322>this <00:09:08.483>further.

00:09:08.590 --> 00:09:08.600 align:start position:0%
further.

00:09:08.600 --> 00:09:11.230 align:start position:0%
<00:09:08.600>further. <00:09:08.755>And <00:09:08.911>by <00:09:09.066>the <00:09:09.222>way, <00:09:09.377>you <00:09:09.533>can <00:09:09.688>see <00:09:09.844>all <00:09:10.000>the

00:09:11.230 --> 00:09:11.240 align:start position:0%
<00:09:11.230>And <00:09:11.231>by <00:09:11.232>the <00:09:11.233>way, <00:09:11.234>you <00:09:11.235>can <00:09:11.236>see <00:09:11.237>all <00:09:11.239>the

00:09:11.240 --> 00:09:13.910 align:start position:0%
<00:09:11.240>And <00:09:11.333>by <00:09:11.426>the <00:09:11.520>way, <00:09:11.613>you <00:09:11.706>can <00:09:11.800>see <00:09:11.893>all <00:09:11.986>the <00:09:12.080>quotes <00:09:12.173>here <00:09:12.266>um <00:09:12.360>are <00:09:12.453>anonymized, <00:09:12.546>but <00:09:12.640>you

00:09:13.910 --> 00:09:13.920 align:start position:0%
<00:09:13.910>quotes <00:09:13.911>here <00:09:13.913>um <00:09:13.914>are <00:09:13.915>anonymized, <00:09:13.917>but <00:09:13.919>you

00:09:13.920 --> 00:09:15.470 align:start position:0%
<00:09:13.920>quotes <00:09:14.036>here <00:09:14.152>um <00:09:14.268>are <00:09:14.385>anonymized, <00:09:14.501>but <00:09:14.617>you <00:09:14.733>can <00:09:14.850>see <00:09:14.966>people <00:09:15.082>are <00:09:15.198>complaining <00:09:15.315>like,

00:09:15.470 --> 00:09:15.480 align:start position:0%
<00:09:15.470>can <00:09:15.471>see <00:09:15.473>people <00:09:15.475>are <00:09:15.477>complaining <00:09:15.479>like,

00:09:15.480 --> 00:09:17.829 align:start position:0%
<00:09:15.480>can <00:09:15.607>see <00:09:15.734>people <00:09:15.861>are <00:09:15.989>complaining <00:09:16.116>like, <00:09:16.243>"What <00:09:16.370>happened <00:09:16.498>with <00:09:16.625>40 <00:09:16.752>million <00:09:16.880>tokens?

00:09:17.829 --> 00:09:17.839 align:start position:0%
<00:09:17.829>"What <00:09:17.830>happened <00:09:17.832>with <00:09:17.834>40 <00:09:17.836>million <00:09:17.838>tokens?

00:09:17.839 --> 00:09:21.710 align:start position:0%
<00:09:17.839>"What <00:09:17.955>happened <00:09:18.072>with <00:09:18.189>40 <00:09:18.305>million <00:09:18.422>tokens? <00:09:18.539>Uh <00:09:18.655>why <00:09:18.772>this <00:09:18.888>investigation <00:09:19.005>took <00:09:19.122>so <00:09:19.238>long?"

00:09:21.710 --> 00:09:21.720 align:start position:0%
<00:09:21.710>Uh <00:09:21.711>why <00:09:21.713>this <00:09:21.714>investigation <00:09:21.716>took <00:09:21.717>so <00:09:21.719>long?"

00:09:21.720 --> 00:09:26.190 align:start position:0%
<00:09:21.720>Uh <00:09:21.920>why <00:09:22.120>this <00:09:22.320>investigation <00:09:22.520>took <00:09:22.720>so <00:09:22.920>long?" <00:09:23.120>So,

00:09:26.200 --> 00:09:28.390 align:start position:0%
<00:09:26.200>when <00:09:26.433>our <00:09:26.666>one <00:09:26.900>of <00:09:27.133>our <00:09:27.366>research <00:09:27.600>engineers

00:09:28.390 --> 00:09:28.400 align:start position:0%
<00:09:28.390>when <00:09:28.391>our <00:09:28.393>one <00:09:28.394>of <00:09:28.396>our <00:09:28.397>research <00:09:28.399>engineers

00:09:28.400 --> 00:09:31.270 align:start position:0%
<00:09:28.400>when <00:09:28.555>our <00:09:28.711>one <00:09:28.866>of <00:09:29.022>our <00:09:29.177>research <00:09:29.333>engineers <00:09:29.488>investigated <00:09:29.644>it, <00:09:29.800>um

00:09:31.270 --> 00:09:31.280 align:start position:0%
<00:09:31.270>investigated <00:09:31.274>it, <00:09:31.279>um

00:09:31.280 --> 00:09:33.150 align:start position:0%
<00:09:31.280>investigated <00:09:31.454>it, <00:09:31.630>um <00:09:31.805>he <00:09:31.980>did <00:09:32.155>notice <00:09:32.329>something <00:09:32.505>very <00:09:32.680>very

00:09:33.150 --> 00:09:33.160 align:start position:0%
<00:09:33.150>he <00:09:33.151>did <00:09:33.153>notice <00:09:33.155>something <00:09:33.157>very <00:09:33.159>very

00:09:33.160 --> 00:09:34.430 align:start position:0%
<00:09:33.160>he <00:09:33.350>did <00:09:33.540>notice <00:09:33.731>something <00:09:33.921>very <00:09:34.112>very <00:09:34.303>interesting.

00:09:34.430 --> 00:09:34.440 align:start position:0%
interesting.

00:09:34.440 --> 00:09:37.510 align:start position:0%
<00:09:34.440>interesting. <00:09:34.615>Our <00:09:34.790>when <00:09:34.965>we <00:09:35.140>look <00:09:35.315>on <00:09:35.490>the <00:09:35.665>distribution <00:09:35.840>of

00:09:37.510 --> 00:09:37.520 align:start position:0%
<00:09:37.510>Our <00:09:37.511>when <00:09:37.512>we <00:09:37.513>look <00:09:37.515>on <00:09:37.516>the <00:09:37.517>distribution <00:09:37.519>of

00:09:37.520 --> 00:09:39.470 align:start position:0%
<00:09:37.520>Our <00:09:37.636>when <00:09:37.753>we <00:09:37.870>look <00:09:37.986>on <00:09:38.103>the <00:09:38.220>distribution <00:09:38.336>of <00:09:38.453>the <00:09:38.569>tokens, <00:09:38.686>so <00:09:38.803>you <00:09:38.920>have

00:09:39.470 --> 00:09:39.480 align:start position:0%
<00:09:39.470>the <00:09:39.472>tokens, <00:09:39.474>so <00:09:39.476>you <00:09:39.479>have

00:09:39.480 --> 00:09:41.550 align:start position:0%
<00:09:39.480>the <00:09:39.655>tokens, <00:09:39.830>so <00:09:40.005>you <00:09:40.180>have <00:09:40.355>input <00:09:40.530>tokens, <00:09:40.705>output <00:09:40.880>tokens,

00:09:41.550 --> 00:09:41.560 align:start position:0%
<00:09:41.550>input <00:09:41.553>tokens, <00:09:41.555>output <00:09:41.559>tokens,

00:09:41.560 --> 00:09:43.870 align:start position:0%
<00:09:41.560>input <00:09:41.699>tokens, <00:09:41.839>output <00:09:41.979>tokens, <00:09:42.119>but <00:09:42.260>there <00:09:42.400>is <00:09:42.540>another <00:09:42.680>type <00:09:42.819>of <00:09:42.959>tokens.

00:09:43.870 --> 00:09:43.880 align:start position:0%
<00:09:43.870>but <00:09:43.871>there <00:09:43.873>is <00:09:43.874>another <00:09:43.876>type <00:09:43.877>of <00:09:43.879>tokens.

00:09:43.880 --> 00:09:48.510 align:start position:0%
<00:09:43.880>but <00:09:44.020>there <00:09:44.160>is <00:09:44.300>another <00:09:44.439>type <00:09:44.580>of <00:09:44.720>tokens. <00:09:44.860>Anyone <00:09:45.000>knows <00:09:45.140>which <00:09:45.280>one?

00:09:48.520 --> 00:09:50.030 align:start position:0%
<00:09:48.520>The <00:09:48.791>context. <00:09:49.063>Okay, <00:09:49.335>I'm <00:09:49.607>talking <00:09:49.879>about

00:09:50.030 --> 00:09:50.040 align:start position:0%
<00:09:50.030>The <00:09:50.031>context. <00:09:50.033>Okay, <00:09:50.035>I'm <00:09:50.037>talking <00:09:50.039>about

00:09:50.040 --> 00:09:51.470 align:start position:0%
<00:09:50.040>The <00:09:50.156>context. <00:09:50.274>Okay, <00:09:50.391>I'm <00:09:50.508>talking <00:09:50.625>about <00:09:50.742>like <00:09:50.859>there's <00:09:50.976>input, <00:09:51.093>output. <00:09:51.210>If <00:09:51.327>you'll

00:09:51.470 --> 00:09:51.480 align:start position:0%
<00:09:51.470>like <00:09:51.471>there's <00:09:51.473>input, <00:09:51.475>output. <00:09:51.477>If <00:09:51.479>you'll

00:09:51.480 --> 00:09:54.430 align:start position:0%
<00:09:51.480>like <00:09:51.607>there's <00:09:51.734>input, <00:09:51.861>output. <00:09:51.989>If <00:09:52.116>you'll <00:09:52.243>open <00:09:52.370>cloud <00:09:52.498>pricing <00:09:52.625>or <00:09:52.752>so <00:09:52.880>on.

00:09:54.430 --> 00:09:54.440 align:start position:0%
<00:09:54.430>open <00:09:54.431>cloud <00:09:54.433>pricing <00:09:54.435>or <00:09:54.437>so <00:09:54.439>on.

00:09:54.440 --> 00:09:57.310 align:start position:0%
<00:09:54.440>open <00:09:54.567>cloud <00:09:54.694>pricing <00:09:54.821>or <00:09:54.949>so <00:09:55.076>on. <00:09:55.203>Cached. <00:09:55.330>Okay, <00:09:55.458>exactly. <00:09:55.585>Cached <00:09:55.712>tokens. <00:09:55.840>Do

00:09:57.310 --> 00:09:57.320 align:start position:0%
<00:09:57.310>Cached. <00:09:57.311>Okay, <00:09:57.313>exactly. <00:09:57.315>Cached <00:09:57.317>tokens. <00:09:57.319>Do

00:09:57.320 --> 00:09:59.550 align:start position:0%
<00:09:57.320>Cached. <00:09:57.427>Okay, <00:09:57.535>exactly. <00:09:57.643>Cached <00:09:57.750>tokens. <00:09:57.858>Do <00:09:57.966>you <00:09:58.073>know <00:09:58.181>how <00:09:58.289>much <00:09:58.396>cached <00:09:58.504>tokens <00:09:58.612>are <00:09:58.720>more

00:09:59.550 --> 00:09:59.560 align:start position:0%
<00:09:59.550>you <00:09:59.551>know <00:09:59.552>how <00:09:59.553>much <00:09:59.555>cached <00:09:59.556>tokens <00:09:59.557>are <00:09:59.559>more

00:09:59.560 --> 00:10:01.870 align:start position:0%
<00:09:59.560>you <00:09:59.667>know <00:09:59.775>how <00:09:59.883>much <00:09:59.990>cached <00:10:00.098>tokens <00:10:00.206>are <00:10:00.313>more <00:10:00.421>are <00:10:00.529>cheaper <00:10:00.636>than <00:10:00.744>than <00:10:00.852>regular <00:10:00.959>input

00:10:01.870 --> 00:10:01.880 align:start position:0%
<00:10:01.870>are <00:10:01.871>cheaper <00:10:01.873>than <00:10:01.875>than <00:10:01.877>regular <00:10:01.879>input

00:10:01.880 --> 00:10:02.950 align:start position:0%
<00:10:01.880>are <00:10:02.040>cheaper <00:10:02.201>than <00:10:02.361>than <00:10:02.522>regular <00:10:02.682>input <00:10:02.843>tokens?

00:10:02.950 --> 00:10:02.960 align:start position:0%
tokens?

00:10:02.960 --> 00:10:06.110 align:start position:0%
<00:10:02.960>tokens? <00:10:03.193>Usually <00:10:03.426>it's <00:10:03.660>about <00:10:03.893>10 <00:10:04.126>times. <00:10:04.360>Correct.

00:10:06.110 --> 00:10:06.120 align:start position:0%
<00:10:06.110>Usually <00:10:06.111>it's <00:10:06.113>about <00:10:06.115>10 <00:10:06.117>times. <00:10:06.119>Correct.

00:10:06.120 --> 00:10:08.870 align:start position:0%
<00:10:06.120>Usually <00:10:06.227>it's <00:10:06.335>about <00:10:06.443>10 <00:10:06.550>times. <00:10:06.658>Correct. <00:10:06.766>So, <00:10:06.873>10 <00:10:06.981>times <00:10:07.089>is <00:10:07.196>a <00:10:07.304>lot <00:10:07.412>of <00:10:07.520>money.

00:10:08.870 --> 00:10:08.880 align:start position:0%
<00:10:08.870>So, <00:10:08.871>10 <00:10:08.872>times <00:10:08.873>is <00:10:08.875>a <00:10:08.876>lot <00:10:08.877>of <00:10:08.879>money.

00:10:08.880 --> 00:10:10.030 align:start position:0%
<00:10:08.880>So, <00:10:09.009>10 <00:10:09.138>times <00:10:09.268>is <00:10:09.397>a <00:10:09.526>lot <00:10:09.656>of <00:10:09.785>money. <00:10:09.915>Okay?

00:10:10.030 --> 00:10:10.040 align:start position:0%
Okay?

00:10:10.040 --> 00:10:12.510 align:start position:0%
<00:10:10.040>Okay? <00:10:10.195>Uh <00:10:10.351>and <00:10:10.506>what <00:10:10.662>we <00:10:10.817>realized <00:10:10.973>is <00:10:11.128>that <00:10:11.284>when <00:10:11.439>we

00:10:12.510 --> 00:10:12.520 align:start position:0%
<00:10:12.510>Uh <00:10:12.511>and <00:10:12.512>what <00:10:12.513>we <00:10:12.514>realized <00:10:12.515>is <00:10:12.516>that <00:10:12.517>when <00:10:12.519>we

00:10:12.520 --> 00:10:14.750 align:start position:0%
<00:10:12.520>Uh <00:10:12.613>and <00:10:12.706>what <00:10:12.800>we <00:10:12.893>realized <00:10:12.986>is <00:10:13.079>that <00:10:13.173>when <00:10:13.266>we <00:10:13.360>looked <00:10:13.453>not <00:10:13.546>on <00:10:13.640>how <00:10:13.733>many <00:10:13.826>tokens <00:10:13.920>in

00:10:14.750 --> 00:10:14.760 align:start position:0%
<00:10:14.750>looked <00:10:14.751>not <00:10:14.753>on <00:10:14.754>how <00:10:14.756>many <00:10:14.757>tokens <00:10:14.759>in

00:10:14.760 --> 00:10:16.430 align:start position:0%
<00:10:14.760>looked <00:10:14.887>not <00:10:15.014>on <00:10:15.141>how <00:10:15.269>many <00:10:15.396>tokens <00:10:15.523>in <00:10:15.650>general, <00:10:15.778>but <00:10:15.905>actually <00:10:16.032>the <00:10:16.160>distribution

00:10:16.430 --> 00:10:16.440 align:start position:0%
<00:10:16.430>general, <00:10:16.432>but <00:10:16.434>actually <00:10:16.436>the <00:10:16.439>distribution

00:10:16.440 --> 00:10:17.870 align:start position:0%
<00:10:16.440>general, <00:10:16.623>but <00:10:16.807>actually <00:10:16.991>the <00:10:17.175>distribution <00:10:17.359>of <00:10:17.543>token <00:10:17.727>types,

00:10:17.870 --> 00:10:17.880 align:start position:0%
<00:10:17.870>of <00:10:17.874>token <00:10:17.879>types,

00:10:17.880 --> 00:10:20.630 align:start position:0%
<00:10:17.880>of <00:10:18.055>token <00:10:18.230>types, <00:10:18.405>input, <00:10:18.580>cache <00:10:18.755>write, <00:10:18.930>cache <00:10:19.105>read, <00:10:19.280>output

00:10:20.630 --> 00:10:20.640 align:start position:0%
<00:10:20.630>input, <00:10:20.631>cache <00:10:20.633>write, <00:10:20.635>cache <00:10:20.637>read, <00:10:20.639>output

00:10:20.640 --> 00:10:21.710 align:start position:0%
<00:10:20.640>input, <00:10:20.800>cache <00:10:20.961>write, <00:10:21.121>cache <00:10:21.282>read, <00:10:21.442>output <00:10:21.603>tokens,

00:10:21.710 --> 00:10:21.720 align:start position:0%
tokens,

00:10:21.720 --> 00:10:25.110 align:start position:0%
<00:10:21.720>tokens, <00:10:21.953>that's <00:10:22.186>when <00:10:22.420>things <00:10:22.653>started <00:10:22.886>to <00:10:23.120>click

00:10:25.110 --> 00:10:25.120 align:start position:0%
<00:10:25.110>that's <00:10:25.111>when <00:10:25.113>things <00:10:25.115>started <00:10:25.117>to <00:10:25.119>click

00:10:25.120 --> 00:10:26.750 align:start position:0%
<00:10:25.120>that's <00:10:25.295>when <00:10:25.470>things <00:10:25.645>started <00:10:25.820>to <00:10:25.995>click <00:10:26.170>to <00:10:26.345>click, <00:10:26.520>sorry.

00:10:26.750 --> 00:10:26.760 align:start position:0%
<00:10:26.750>to <00:10:26.754>click, <00:10:26.759>sorry.

00:10:26.760 --> 00:10:28.630 align:start position:0%
<00:10:26.760>to <00:10:26.960>click, <00:10:27.160>sorry. <00:10:27.360>And <00:10:27.560>that's <00:10:27.760>when <00:10:27.960>we <00:10:28.160>realized

00:10:28.630 --> 00:10:28.640 align:start position:0%
<00:10:28.630>And <00:10:28.632>that's <00:10:28.634>when <00:10:28.636>we <00:10:28.639>realized

00:10:28.640 --> 00:10:32.470 align:start position:0%
<00:10:28.640>And <00:10:28.814>that's <00:10:28.990>when <00:10:29.165>we <00:10:29.340>realized <00:10:29.515>that <00:10:29.689>for <00:10:29.865>some <00:10:30.040>reason

00:10:32.470 --> 00:10:32.480 align:start position:0%
<00:10:32.470>that <00:10:32.473>for <00:10:32.476>some <00:10:32.479>reason

00:10:32.480 --> 00:10:34.870 align:start position:0%
<00:10:32.480>that <00:10:32.607>for <00:10:32.734>some <00:10:32.861>reason <00:10:32.989>a <00:10:33.116>lot <00:10:33.243>of <00:10:33.370>tokens <00:10:33.498>are <00:10:33.625>not <00:10:33.752>being <00:10:33.880>cached.

00:10:34.870 --> 00:10:34.880 align:start position:0%
<00:10:34.870>a <00:10:34.871>lot <00:10:34.872>of <00:10:34.873>tokens <00:10:34.875>are <00:10:34.876>not <00:10:34.877>being <00:10:34.879>cached.

00:10:34.880 --> 00:10:36.230 align:start position:0%
<00:10:34.880>a <00:10:34.966>lot <00:10:35.053>of <00:10:35.140>tokens <00:10:35.227>are <00:10:35.313>not <00:10:35.400>being <00:10:35.487>cached. <00:10:35.574>Um <00:10:35.661>and <00:10:35.747>I <00:10:35.834>think <00:10:35.921>it's <00:10:36.008>interesting <00:10:36.095>because

00:10:36.230 --> 00:10:36.240 align:start position:0%
<00:10:36.230>Um <00:10:36.231>and <00:10:36.233>I <00:10:36.234>think <00:10:36.236>it's <00:10:36.237>interesting <00:10:36.239>because

00:10:36.240 --> 00:10:38.110 align:start position:0%
<00:10:36.240>Um <00:10:36.356>and <00:10:36.473>I <00:10:36.590>think <00:10:36.706>it's <00:10:36.823>interesting <00:10:36.940>because <00:10:37.056>that's <00:10:37.173>something <00:10:37.290>that <00:10:37.406>has <00:10:37.523>changed <00:10:37.640>over

00:10:38.110 --> 00:10:38.120 align:start position:0%
<00:10:38.110>that's <00:10:38.111>something <00:10:38.113>that <00:10:38.115>has <00:10:38.117>changed <00:10:38.119>over

00:10:38.120 --> 00:10:41.190 align:start position:0%
<00:10:38.120>that's <00:10:38.227>something <00:10:38.335>that <00:10:38.443>has <00:10:38.550>changed <00:10:38.658>over <00:10:38.766>the <00:10:38.873>last <00:10:38.981>year <00:10:39.089>or <00:10:39.196>two <00:10:39.304>years <00:10:39.412>with <00:10:39.520>the

00:10:41.190 --> 00:10:41.200 align:start position:0%
<00:10:41.190>the <00:10:41.191>last <00:10:41.192>year <00:10:41.193>or <00:10:41.195>two <00:10:41.196>years <00:10:41.197>with <00:10:41.199>the

00:10:41.200 --> 00:10:42.750 align:start position:0%
<00:10:41.200>the <00:10:41.316>last <00:10:41.432>year <00:10:41.548>or <00:10:41.665>two <00:10:41.781>years <00:10:41.897>with <00:10:42.013>the <00:10:42.130>abstraction <00:10:42.246>and <00:10:42.362>frameworks <00:10:42.478>that <00:10:42.595>we've

00:10:42.750 --> 00:10:42.760 align:start position:0%
<00:10:42.750>abstraction <00:10:42.752>and <00:10:42.754>frameworks <00:10:42.756>that <00:10:42.759>we've

00:10:42.760 --> 00:10:45.350 align:start position:0%
<00:10:42.760>abstraction <00:10:42.876>and <00:10:42.993>frameworks <00:10:43.110>that <00:10:43.226>we've <00:10:43.343>got. <00:10:43.460>So, <00:10:43.576>two <00:10:43.693>years <00:10:43.810>ago, <00:10:43.926>when <00:10:44.043>you <00:10:44.160>built

00:10:45.350 --> 00:10:45.360 align:start position:0%
<00:10:45.350>got. <00:10:45.351>So, <00:10:45.352>two <00:10:45.353>years <00:10:45.355>ago, <00:10:45.356>when <00:10:45.357>you <00:10:45.359>built

00:10:45.360 --> 00:10:46.870 align:start position:0%
<00:10:45.360>got. <00:10:45.464>So, <00:10:45.569>two <00:10:45.673>years <00:10:45.778>ago, <00:10:45.882>when <00:10:45.987>you <00:10:46.091>built <00:10:46.196>an <00:10:46.300>agent, <00:10:46.405>you <00:10:46.509>would <00:10:46.614>actually <00:10:46.719>implement

00:10:46.870 --> 00:10:46.880 align:start position:0%
<00:10:46.870>an <00:10:46.871>agent, <00:10:46.873>you <00:10:46.875>would <00:10:46.877>actually <00:10:46.879>implement

00:10:46.880 --> 00:10:48.670 align:start position:0%
<00:10:46.880>an <00:10:46.980>agent, <00:10:47.080>you <00:10:47.180>would <00:10:47.280>actually <00:10:47.380>implement <00:10:47.480>the <00:10:47.580>agent <00:10:47.680>loop <00:10:47.780>on <00:10:47.880>your <00:10:47.980>own. <00:10:48.080>The <00:10:48.180>call <00:10:48.280>to

00:10:48.670 --> 00:10:48.680 align:start position:0%
<00:10:48.670>the <00:10:48.671>agent <00:10:48.672>loop <00:10:48.673>on <00:10:48.674>your <00:10:48.675>own. <00:10:48.676>The <00:10:48.677>call <00:10:48.679>to

00:10:48.680 --> 00:10:51.270 align:start position:0%
<00:10:48.680>the <00:10:48.767>agent <00:10:48.854>loop <00:10:48.942>on <00:10:49.030>your <00:10:49.117>own. <00:10:49.204>The <00:10:49.292>call <00:10:49.380>to <00:10:49.467>the <00:10:49.555>LLM, <00:10:49.642>the <00:10:49.729>uh <00:10:49.817>managing <00:10:49.905>the <00:10:49.992>tool <00:10:50.079>calls

00:10:51.270 --> 00:10:51.280 align:start position:0%
<00:10:51.270>the <00:10:51.271>LLM, <00:10:51.272>the <00:10:51.273>uh <00:10:51.275>managing <00:10:51.276>the <00:10:51.277>tool <00:10:51.279>calls

00:10:51.280 --> 00:10:53.470 align:start position:0%
<00:10:51.280>the <00:10:51.387>LLM, <00:10:51.495>the <00:10:51.603>uh <00:10:51.710>managing <00:10:51.818>the <00:10:51.926>tool <00:10:52.033>calls <00:10:52.141>and <00:10:52.249>the <00:10:52.356>responses <00:10:52.464>and <00:10:52.572>so <00:10:52.680>on.

00:10:53.470 --> 00:10:53.480 align:start position:0%
<00:10:53.470>and <00:10:53.471>the <00:10:53.473>responses <00:10:53.475>and <00:10:53.477>so <00:10:53.479>on.

00:10:53.480 --> 00:10:54.550 align:start position:0%
<00:10:53.480>and <00:10:53.640>the <00:10:53.801>responses <00:10:53.961>and <00:10:54.122>so <00:10:54.282>on. <00:10:54.443>Um

00:10:54.550 --> 00:10:54.560 align:start position:0%
Um

00:10:54.560 --> 00:10:57.270 align:start position:0%
<00:10:54.560>Um <00:10:54.734>but <00:10:54.910>with <00:10:55.084>the <00:10:55.260>with <00:10:55.435>the <00:10:55.609>evolvement <00:10:55.785>of <00:10:55.959>the

00:10:57.270 --> 00:10:57.280 align:start position:0%
<00:10:57.270>but <00:10:57.271>with <00:10:57.272>the <00:10:57.273>with <00:10:57.275>the <00:10:57.276>evolvement <00:10:57.277>of <00:10:57.279>the

00:10:57.280 --> 00:10:59.630 align:start position:0%
<00:10:57.280>but <00:10:57.373>with <00:10:57.466>the <00:10:57.560>with <00:10:57.653>the <00:10:57.746>evolvement <00:10:57.839>of <00:10:57.933>the <00:10:58.026>space <00:10:58.120>and <00:10:58.213>with <00:10:58.306>more <00:10:58.400>and <00:10:58.493>more <00:10:58.586>SDK <00:10:58.680>and

00:10:59.630 --> 00:10:59.640 align:start position:0%
<00:10:59.630>space <00:10:59.631>and <00:10:59.632>with <00:10:59.633>more <00:10:59.635>and <00:10:59.636>more <00:10:59.637>SDK <00:10:59.639>and

00:10:59.640 --> 00:11:01.790 align:start position:0%
<00:10:59.640>space <00:10:59.747>and <00:10:59.855>with <00:10:59.963>more <00:11:00.070>and <00:11:00.178>more <00:11:00.286>SDK <00:11:00.393>and <00:11:00.501>agentic <00:11:00.609>SDK, <00:11:00.716>whether <00:11:00.824>it's <00:11:00.932>OpenAI <00:11:01.040>agent

00:11:01.790 --> 00:11:01.800 align:start position:0%
<00:11:01.790>agentic <00:11:01.791>SDK, <00:11:01.793>whether <00:11:01.795>it's <00:11:01.797>OpenAI <00:11:01.799>agent

00:11:01.800 --> 00:11:04.190 align:start position:0%
<00:11:01.800>agentic <00:11:01.916>SDK, <00:11:02.033>whether <00:11:02.150>it's <00:11:02.266>OpenAI <00:11:02.383>agent <00:11:02.500>SDK <00:11:02.616>or <00:11:02.733>Anthropic <00:11:02.849>Claude's <00:11:02.966>agent <00:11:03.083>SDK <00:11:03.199>and

00:11:04.190 --> 00:11:04.200 align:start position:0%
<00:11:04.190>SDK <00:11:04.191>or <00:11:04.193>Anthropic <00:11:04.194>Claude's <00:11:04.196>agent <00:11:04.197>SDK <00:11:04.199>and

00:11:04.200 --> 00:11:07.110 align:start position:0%
<00:11:04.200>SDK <00:11:04.307>or <00:11:04.415>Anthropic <00:11:04.523>Claude's <00:11:04.630>agent <00:11:04.738>SDK <00:11:04.846>and <00:11:04.953>so <00:11:05.061>on, <00:11:05.169>um <00:11:05.276>you <00:11:05.384>have <00:11:05.492>this <00:11:05.600>abstraction.

00:11:07.110 --> 00:11:07.120 align:start position:0%
<00:11:07.110>so <00:11:07.111>on, <00:11:07.113>um <00:11:07.114>you <00:11:07.116>have <00:11:07.117>this <00:11:07.119>abstraction.

00:11:07.120 --> 00:11:08.590 align:start position:0%
<00:11:07.120>so <00:11:07.221>on, <00:11:07.323>um <00:11:07.425>you <00:11:07.527>have <00:11:07.628>this <00:11:07.730>abstraction. <00:11:07.832>And <00:11:07.934>when <00:11:08.035>you <00:11:08.137>have <00:11:08.239>the <00:11:08.341>abstraction, <00:11:08.443>you

00:11:08.590 --> 00:11:08.600 align:start position:0%
<00:11:08.590>And <00:11:08.591>when <00:11:08.593>you <00:11:08.594>have <00:11:08.596>the <00:11:08.597>abstraction, <00:11:08.599>you

00:11:08.600 --> 00:11:10.190 align:start position:0%
<00:11:08.600>And <00:11:08.707>when <00:11:08.815>you <00:11:08.923>have <00:11:09.030>the <00:11:09.138>abstraction, <00:11:09.246>you <00:11:09.353>have <00:11:09.461>no <00:11:09.569>idea <00:11:09.676>what's <00:11:09.784>happening <00:11:09.892>behind <00:11:10.000>the

00:11:10.190 --> 00:11:10.200 align:start position:0%
<00:11:10.190>have <00:11:10.191>no <00:11:10.193>idea <00:11:10.194>what's <00:11:10.196>happening <00:11:10.197>behind <00:11:10.199>the

00:11:10.200 --> 00:11:13.150 align:start position:0%
<00:11:10.200>have <00:11:10.307>no <00:11:10.415>idea <00:11:10.523>what's <00:11:10.630>happening <00:11:10.738>behind <00:11:10.846>the <00:11:10.953>scenes. <00:11:11.061>And <00:11:11.169>on <00:11:11.276>one <00:11:11.384>hand, <00:11:11.492>it's <00:11:11.600>great

00:11:13.150 --> 00:11:13.160 align:start position:0%
<00:11:13.150>scenes. <00:11:13.151>And <00:11:13.153>on <00:11:13.154>one <00:11:13.156>hand, <00:11:13.157>it's <00:11:13.159>great

00:11:13.160 --> 00:11:14.910 align:start position:0%
<00:11:13.160>scenes. <00:11:13.267>And <00:11:13.375>on <00:11:13.483>one <00:11:13.590>hand, <00:11:13.698>it's <00:11:13.806>great <00:11:13.913>because <00:11:14.021>it <00:11:14.129>allows <00:11:14.236>us <00:11:14.344>to <00:11:14.452>develop <00:11:14.560>much

00:11:14.910 --> 00:11:14.920 align:start position:0%
<00:11:14.910>because <00:11:14.911>it <00:11:14.913>allows <00:11:14.914>us <00:11:14.915>to <00:11:14.917>develop <00:11:14.919>much

00:11:14.920 --> 00:11:18.190 align:start position:0%
<00:11:14.920>because <00:11:15.020>it <00:11:15.120>allows <00:11:15.219>us <00:11:15.319>to <00:11:15.420>develop <00:11:15.520>much <00:11:15.620>faster, <00:11:15.719>right? <00:11:15.819>But <00:11:15.920>on <00:11:16.020>the <00:11:16.120>other <00:11:16.219>hand, <00:11:16.319>it

00:11:18.190 --> 00:11:18.200 align:start position:0%
<00:11:18.190>faster, <00:11:18.191>right? <00:11:18.192>But <00:11:18.193>on <00:11:18.195>the <00:11:18.196>other <00:11:18.197>hand, <00:11:18.199>it

00:11:18.200 --> 00:11:20.230 align:start position:0%
<00:11:18.200>faster, <00:11:18.307>right? <00:11:18.415>But <00:11:18.523>on <00:11:18.630>the <00:11:18.738>other <00:11:18.846>hand, <00:11:18.953>it <00:11:19.061>makes <00:11:19.169>us <00:11:19.276>sometimes <00:11:19.384>make <00:11:19.492>mistakes <00:11:19.600>we

00:11:20.230 --> 00:11:20.240 align:start position:0%
<00:11:20.230>makes <00:11:20.231>us <00:11:20.233>sometimes <00:11:20.235>make <00:11:20.237>mistakes <00:11:20.239>we

00:11:20.240 --> 00:11:22.070 align:start position:0%
<00:11:20.240>makes <00:11:20.367>us <00:11:20.494>sometimes <00:11:20.621>make <00:11:20.749>mistakes <00:11:20.876>we <00:11:21.003>don't <00:11:21.130>even <00:11:21.258>realize. <00:11:21.385>And <00:11:21.512>that's <00:11:21.640>what

00:11:22.070 --> 00:11:22.080 align:start position:0%
<00:11:22.070>don't <00:11:22.071>even <00:11:22.073>realize. <00:11:22.075>And <00:11:22.077>that's <00:11:22.079>what

00:11:22.080 --> 00:11:23.590 align:start position:0%
<00:11:22.080>don't <00:11:22.274>even <00:11:22.468>realize. <00:11:22.662>And <00:11:22.856>that's <00:11:23.050>what <00:11:23.244>happened <00:11:23.439>here.

00:11:23.590 --> 00:11:23.600 align:start position:0%
<00:11:23.590>happened <00:11:23.599>here.

00:11:23.600 --> 00:11:25.710 align:start position:0%
<00:11:23.600>happened <00:11:23.800>here. <00:11:24.000>So, <00:11:24.200>only <00:11:24.400>after <00:11:24.600>investigation <00:11:24.800>we <00:11:25.000>realized

00:11:25.710 --> 00:11:25.720 align:start position:0%
<00:11:25.710>So, <00:11:25.711>only <00:11:25.713>after <00:11:25.715>investigation <00:11:25.717>we <00:11:25.719>realized

00:11:25.720 --> 00:11:27.870 align:start position:0%
<00:11:25.720>So, <00:11:25.836>only <00:11:25.953>after <00:11:26.070>investigation <00:11:26.186>we <00:11:26.303>realized <00:11:26.420>that <00:11:26.536>behind <00:11:26.653>the <00:11:26.770>scenes, <00:11:26.886>the <00:11:27.003>way <00:11:27.120>our

00:11:27.870 --> 00:11:27.880 align:start position:0%
<00:11:27.870>that <00:11:27.871>behind <00:11:27.873>the <00:11:27.874>scenes, <00:11:27.876>the <00:11:27.877>way <00:11:27.879>our

00:11:27.880 --> 00:11:31.270 align:start position:0%
<00:11:27.880>that <00:11:27.996>behind <00:11:28.113>the <00:11:28.230>scenes, <00:11:28.346>the <00:11:28.463>way <00:11:28.580>our <00:11:28.696>agent <00:11:28.813>SDK <00:11:28.930>actually <00:11:29.046>manages <00:11:29.163>and <00:11:29.280>utilizes

00:11:31.270 --> 00:11:31.280 align:start position:0%
<00:11:31.270>agent <00:11:31.271>SDK <00:11:31.273>actually <00:11:31.275>manages <00:11:31.277>and <00:11:31.279>utilizes

00:11:31.280 --> 00:11:33.430 align:start position:0%
<00:11:31.280>agent <00:11:31.396>SDK <00:11:31.513>actually <00:11:31.630>manages <00:11:31.746>and <00:11:31.863>utilizes <00:11:31.980>the <00:11:32.096>system <00:11:32.213>prompt <00:11:32.329>and <00:11:32.446>tool <00:11:32.563>calls <00:11:32.680>and

00:11:33.430 --> 00:11:33.440 align:start position:0%
<00:11:33.430>the <00:11:33.431>system <00:11:33.433>prompt <00:11:33.434>and <00:11:33.436>tool <00:11:33.437>calls <00:11:33.439>and

00:11:33.440 --> 00:11:36.110 align:start position:0%
<00:11:33.440>the <00:11:33.595>system <00:11:33.751>prompt <00:11:33.906>and <00:11:34.062>tool <00:11:34.217>calls <00:11:34.373>and <00:11:34.528>everything <00:11:34.684>we've <00:11:34.840>defined

00:11:36.110 --> 00:11:36.120 align:start position:0%
<00:11:36.110>everything <00:11:36.114>we've <00:11:36.119>defined

00:11:36.120 --> 00:11:41.470 align:start position:0%
<00:11:36.120>everything <00:11:36.295>we've <00:11:36.470>defined <00:11:36.645>was <00:11:36.820>far <00:11:36.995>off <00:11:37.170>from <00:11:37.345>being <00:11:37.520>cost-effective.

00:11:41.480 --> 00:11:43.350 align:start position:0%
So,

00:11:43.350 --> 00:11:43.360 align:start position:0%
So,

00:11:43.360 --> 00:11:45.150 align:start position:0%
<00:11:43.360>So, <00:11:43.826>we <00:11:44.293>improved <00:11:44.760>this

00:11:45.150 --> 00:11:45.160 align:start position:0%
<00:11:45.150>we <00:11:45.154>improved <00:11:45.159>this

00:11:45.160 --> 00:11:47.670 align:start position:0%
<00:11:45.160>we <00:11:45.334>improved <00:11:45.510>this <00:11:45.685>and <00:11:45.860>we <00:11:46.035>got <00:11:46.209>significant <00:11:46.385>improvement. <00:11:46.560>Uh

00:11:47.670 --> 00:11:47.680 align:start position:0%
<00:11:47.670>and <00:11:47.671>we <00:11:47.673>got <00:11:47.675>significant <00:11:47.677>improvement. <00:11:47.679>Uh

00:11:47.680 --> 00:11:50.230 align:start position:0%
<00:11:47.680>and <00:11:47.854>we <00:11:48.030>got <00:11:48.204>significant <00:11:48.380>improvement. <00:11:48.555>Uh <00:11:48.729>I'm <00:11:48.905>talking <00:11:49.079>about

00:11:50.230 --> 00:11:50.240 align:start position:0%
<00:11:50.230>I'm <00:11:50.234>talking <00:11:50.239>about

00:11:50.240 --> 00:11:53.310 align:start position:0%
<00:11:50.240>I'm <00:11:50.380>talking <00:11:50.520>about <00:11:50.660>give <00:11:50.800>or <00:11:50.940>take <00:11:51.080>six, <00:11:51.220>seven <00:11:51.360>times <00:11:51.500>more <00:11:51.640>cash

00:11:53.310 --> 00:11:53.320 align:start position:0%
<00:11:53.310>give <00:11:53.311>or <00:11:53.312>take <00:11:53.313>six, <00:11:53.315>seven <00:11:53.316>times <00:11:53.317>more <00:11:53.319>cash

00:11:53.320 --> 00:11:54.550 align:start position:0%
<00:11:53.320>give <00:11:53.458>or <00:11:53.596>take <00:11:53.735>six, <00:11:53.873>seven <00:11:54.011>times <00:11:54.150>more <00:11:54.288>cash <00:11:54.426>tokens.

00:11:54.550 --> 00:11:54.560 align:start position:0%
tokens.

00:11:54.560 --> 00:11:56.110 align:start position:0%
<00:11:54.560>tokens. <00:11:54.734>So, <00:11:54.908>back <00:11:55.083>to <00:11:55.257>our <00:11:55.431>clients, <00:11:55.606>I <00:11:55.780>think, <00:11:55.955>you

00:11:56.110 --> 00:11:56.120 align:start position:0%
<00:11:56.110>So, <00:11:56.111>back <00:11:56.112>to <00:11:56.113>our <00:11:56.115>clients, <00:11:56.116>I <00:11:56.117>think, <00:11:56.119>you

00:11:56.120 --> 00:11:58.870 align:start position:0%
<00:11:56.120>So, <00:11:56.247>back <00:11:56.374>to <00:11:56.501>our <00:11:56.629>clients, <00:11:56.756>I <00:11:56.883>think, <00:11:57.010>you <00:11:57.138>know, <00:11:57.265>on <00:11:57.392>one <00:11:57.520>hand

00:11:58.870 --> 00:11:58.880 align:start position:0%
<00:11:58.870>know, <00:11:58.873>on <00:11:58.876>one <00:11:58.879>hand

00:11:58.880 --> 00:12:01.430 align:start position:0%
<00:11:58.880>know, <00:11:59.055>on <00:11:59.230>one <00:11:59.405>hand <00:11:59.580>there <00:11:59.755>was <00:11:59.930>a <00:12:00.105>question <00:12:00.280>of

00:12:01.430 --> 00:12:01.440 align:start position:0%
<00:12:01.430>there <00:12:01.432>was <00:12:01.434>a <00:12:01.436>question <00:12:01.439>of

00:12:01.440 --> 00:12:03.270 align:start position:0%
<00:12:01.440>there <00:12:01.567>was <00:12:01.694>a <00:12:01.821>question <00:12:01.949>of <00:12:02.076>why <00:12:02.203>do <00:12:02.330>they <00:12:02.458>complain <00:12:02.585>about <00:12:02.712>many <00:12:02.840>tokens?

00:12:03.270 --> 00:12:03.280 align:start position:0%
<00:12:03.270>why <00:12:03.271>do <00:12:03.273>they <00:12:03.274>complain <00:12:03.276>about <00:12:03.277>many <00:12:03.279>tokens?

00:12:03.280 --> 00:12:04.550 align:start position:0%
<00:12:03.280>why <00:12:03.375>do <00:12:03.470>they <00:12:03.565>complain <00:12:03.661>about <00:12:03.756>many <00:12:03.851>tokens? <00:12:03.946>I <00:12:04.041>think <00:12:04.137>this <00:12:04.232>also <00:12:04.327>alleviates <00:12:04.423>the

00:12:04.550 --> 00:12:04.560 align:start position:0%
<00:12:04.550>I <00:12:04.551>think <00:12:04.553>this <00:12:04.555>also <00:12:04.557>alleviates <00:12:04.559>the

00:12:04.560 --> 00:12:07.350 align:start position:0%
<00:12:04.560>I <00:12:04.676>think <00:12:04.793>this <00:12:04.910>also <00:12:05.026>alleviates <00:12:05.143>the <00:12:05.260>question <00:12:05.376>of <00:12:05.493>how <00:12:05.609>much <00:12:05.726>value <00:12:05.843>they <00:12:05.959>perceive

00:12:07.350 --> 00:12:07.360 align:start position:0%
<00:12:07.350>question <00:12:07.351>of <00:12:07.353>how <00:12:07.354>much <00:12:07.356>value <00:12:07.357>they <00:12:07.359>perceive

00:12:07.360 --> 00:12:08.750 align:start position:0%
<00:12:07.360>question <00:12:07.464>of <00:12:07.568>how <00:12:07.672>much <00:12:07.777>value <00:12:07.881>they <00:12:07.985>perceive <00:12:08.089>they're <00:12:08.194>getting <00:12:08.298>from <00:12:08.402>an <00:12:08.506>the <00:12:08.611>agent,

00:12:08.750 --> 00:12:08.760 align:start position:0%
<00:12:08.750>they're <00:12:08.751>getting <00:12:08.753>from <00:12:08.755>an <00:12:08.757>the <00:12:08.759>agent,

00:12:08.760 --> 00:12:10.750 align:start position:0%
<00:12:08.760>they're <00:12:08.876>getting <00:12:08.993>from <00:12:09.110>an <00:12:09.226>the <00:12:09.343>agent, <00:12:09.460>right? <00:12:09.576>This <00:12:09.693>is <00:12:09.810>also <00:12:09.926>another <00:12:10.043>question. <00:12:10.160>Uh

00:12:10.750 --> 00:12:10.760 align:start position:0%
<00:12:10.750>right? <00:12:10.751>This <00:12:10.753>is <00:12:10.754>also <00:12:10.756>another <00:12:10.757>question. <00:12:10.759>Uh

00:12:10.760 --> 00:12:13.070 align:start position:0%
<00:12:10.760>right? <00:12:10.860>This <00:12:10.960>is <00:12:11.060>also <00:12:11.160>another <00:12:11.260>question. <00:12:11.360>Uh <00:12:11.460>but <00:12:11.560>on <00:12:11.660>the <00:12:11.760>other <00:12:11.860>way <00:12:11.960>around, <00:12:12.060>they <00:12:12.160>were

00:12:13.070 --> 00:12:13.080 align:start position:0%
<00:12:13.070>but <00:12:13.071>on <00:12:13.072>the <00:12:13.073>other <00:12:13.075>way <00:12:13.076>around, <00:12:13.077>they <00:12:13.079>were

00:12:13.080 --> 00:12:14.670 align:start position:0%
<00:12:13.080>but <00:12:13.235>on <00:12:13.391>the <00:12:13.546>other <00:12:13.702>way <00:12:13.857>around, <00:12:14.013>they <00:12:14.168>were <00:12:14.324>actually <00:12:14.480>right.

00:12:14.670 --> 00:12:14.680 align:start position:0%
<00:12:14.670>actually <00:12:14.679>right.

00:12:14.680 --> 00:12:18.070 align:start position:0%
<00:12:14.680>actually <00:12:14.835>right. <00:12:14.991>Our <00:12:15.146>agent <00:12:15.302>was <00:12:15.457>not <00:12:15.613>utilizing <00:12:15.768>cash <00:12:15.924>as <00:12:16.079>good

00:12:18.070 --> 00:12:18.080 align:start position:0%
<00:12:18.070>Our <00:12:18.071>agent <00:12:18.072>was <00:12:18.073>not <00:12:18.075>utilizing <00:12:18.076>cash <00:12:18.077>as <00:12:18.079>good

00:12:18.080 --> 00:12:20.470 align:start position:0%
<00:12:18.080>Our <00:12:18.167>agent <00:12:18.255>was <00:12:18.342>not <00:12:18.430>utilizing <00:12:18.517>cash <00:12:18.605>as <00:12:18.692>good <00:12:18.780>as <00:12:18.867>it <00:12:18.955>should. <00:12:19.042>Uh <00:12:19.130>and <00:12:19.217>in <00:12:19.305>general, <00:12:19.392>as <00:12:19.480>a

00:12:20.470 --> 00:12:20.480 align:start position:0%
<00:12:20.470>as <00:12:20.471>it <00:12:20.472>should. <00:12:20.473>Uh <00:12:20.474>and <00:12:20.475>in <00:12:20.476>general, <00:12:20.477>as <00:12:20.479>a

00:12:20.480 --> 00:12:23.790 align:start position:0%
<00:12:20.480>as <00:12:20.567>it <00:12:20.655>should. <00:12:20.742>Uh <00:12:20.830>and <00:12:20.917>in <00:12:21.005>general, <00:12:21.092>as <00:12:21.180>a <00:12:21.267>benchmark, <00:12:21.355>if <00:12:21.442>you <00:12:21.530>have <00:12:21.617>less <00:12:21.705>than <00:12:21.792>80% <00:12:21.880>of

00:12:23.790 --> 00:12:23.800 align:start position:0%
<00:12:23.790>benchmark, <00:12:23.791>if <00:12:23.792>you <00:12:23.793>have <00:12:23.795>less <00:12:23.796>than <00:12:23.797>80% <00:12:23.799>of

00:12:23.800 --> 00:12:26.950 align:start position:0%
<00:12:23.800>benchmark, <00:12:23.907>if <00:12:24.015>you <00:12:24.123>have <00:12:24.230>less <00:12:24.338>than <00:12:24.446>80% <00:12:24.553>of <00:12:24.661>cash <00:12:24.769>tokens, <00:12:24.876>um <00:12:24.984>usually <00:12:25.092>it <00:12:25.199>means

00:12:26.950 --> 00:12:26.960 align:start position:0%
<00:12:26.950>cash <00:12:26.951>tokens, <00:12:26.953>um <00:12:26.955>usually <00:12:26.957>it <00:12:26.959>means

00:12:26.960 --> 00:12:28.790 align:start position:0%
<00:12:26.960>cash <00:12:27.060>tokens, <00:12:27.160>um <00:12:27.260>usually <00:12:27.360>it <00:12:27.460>means <00:12:27.560>that <00:12:27.660>a <00:12:27.760>lot <00:12:27.860>of <00:12:27.960>tokens <00:12:28.060>are <00:12:28.160>that <00:12:28.260>caching <00:12:28.360>is

00:12:28.790 --> 00:12:28.800 align:start position:0%
<00:12:28.790>that <00:12:28.791>a <00:12:28.792>lot <00:12:28.793>of <00:12:28.794>tokens <00:12:28.795>are <00:12:28.796>that <00:12:28.797>caching <00:12:28.799>is

00:12:28.800 --> 00:12:31.270 align:start position:0%
<00:12:28.800>that <00:12:28.893>a <00:12:28.986>lot <00:12:29.079>of <00:12:29.173>tokens <00:12:29.266>are <00:12:29.359>that <00:12:29.453>caching <00:12:29.546>is <00:12:29.640>not <00:12:29.733>working <00:12:29.826>properly. <00:12:29.920>So, <00:12:30.013>I <00:12:30.106>would <00:12:30.199>urge

00:12:31.270 --> 00:12:31.280 align:start position:0%
<00:12:31.270>not <00:12:31.271>working <00:12:31.273>properly. <00:12:31.274>So, <00:12:31.276>I <00:12:31.277>would <00:12:31.279>urge

00:12:31.280 --> 00:12:33.270 align:start position:0%
<00:12:31.280>not <00:12:31.387>working <00:12:31.495>properly. <00:12:31.603>So, <00:12:31.710>I <00:12:31.818>would <00:12:31.926>urge <00:12:32.033>you <00:12:32.141>to <00:12:32.249>actually <00:12:32.356>go <00:12:32.464>in <00:12:32.572>and <00:12:32.680>investigate

00:12:33.270 --> 00:12:33.280 align:start position:0%
<00:12:33.270>you <00:12:33.271>to <00:12:33.273>actually <00:12:33.274>go <00:12:33.276>in <00:12:33.277>and <00:12:33.279>investigate

00:12:33.280 --> 00:12:35.470 align:start position:0%
<00:12:33.280>you <00:12:33.387>to <00:12:33.495>actually <00:12:33.603>go <00:12:33.710>in <00:12:33.818>and <00:12:33.926>investigate <00:12:34.033>that. <00:12:34.141>Uh <00:12:34.249>for <00:12:34.356>that, <00:12:34.464>obviously, <00:12:34.572>you <00:12:34.680>need

00:12:35.470 --> 00:12:35.480 align:start position:0%
<00:12:35.470>that. <00:12:35.471>Uh <00:12:35.473>for <00:12:35.474>that, <00:12:35.476>obviously, <00:12:35.477>you <00:12:35.479>need

00:12:35.480 --> 00:12:37.950 align:start position:0%
<00:12:35.480>that. <00:12:35.587>Uh <00:12:35.695>for <00:12:35.803>that, <00:12:35.910>obviously, <00:12:36.018>you <00:12:36.126>need <00:12:36.233>to <00:12:36.341>have <00:12:36.449>history <00:12:36.556>of <00:12:36.664>all <00:12:36.772>the <00:12:36.880>conversations

00:12:37.950 --> 00:12:37.960 align:start position:0%
<00:12:37.950>to <00:12:37.951>have <00:12:37.953>history <00:12:37.954>of <00:12:37.956>all <00:12:37.957>the <00:12:37.959>conversations

00:12:37.960 --> 00:12:40.670 align:start position:0%
<00:12:37.960>to <00:12:38.067>have <00:12:38.175>history <00:12:38.283>of <00:12:38.390>all <00:12:38.498>the <00:12:38.606>conversations <00:12:38.713>from <00:12:38.821>production, <00:12:38.929>um <00:12:39.036>full <00:12:39.144>tracing <00:12:39.252>of <00:12:39.360>what

00:12:40.670 --> 00:12:40.680 align:start position:0%
<00:12:40.670>from <00:12:40.671>production, <00:12:40.673>um <00:12:40.674>full <00:12:40.675>tracing <00:12:40.677>of <00:12:40.679>what

00:12:40.680 --> 00:12:42.350 align:start position:0%
<00:12:40.680>from <00:12:40.787>production, <00:12:40.895>um <00:12:41.003>full <00:12:41.110>tracing <00:12:41.218>of <00:12:41.326>what <00:12:41.433>happened <00:12:41.541>behind <00:12:41.649>the <00:12:41.756>scenes, <00:12:41.864>so <00:12:41.972>you <00:12:42.079>can

00:12:42.350 --> 00:12:42.360 align:start position:0%
<00:12:42.350>happened <00:12:42.351>behind <00:12:42.353>the <00:12:42.354>scenes, <00:12:42.356>so <00:12:42.357>you <00:12:42.359>can

00:12:42.360 --> 00:12:47.870 align:start position:0%
<00:12:42.360>happened <00:12:42.476>behind <00:12:42.593>the <00:12:42.710>scenes, <00:12:42.826>so <00:12:42.943>you <00:12:43.060>can <00:12:43.176>actually <00:12:43.293>debug <00:12:43.410>and <00:12:43.526>do <00:12:43.643>these <00:12:43.760>evaluations.

00:12:47.880 --> 00:12:50.150 align:start position:0%
<00:12:47.880>So, <00:12:49.280>um

00:12:50.150 --> 00:12:50.160 align:start position:0%
<00:12:50.150>So, <00:12:50.159>um

00:12:50.160 --> 00:12:52.510 align:start position:0%
<00:12:50.160>So, <00:12:50.334>um <00:12:50.510>we're <00:12:50.685>in <00:12:50.860>GA, <00:12:51.035>Ollie <00:12:51.209>is <00:12:51.385>now <00:12:51.560>quite

00:12:52.510 --> 00:12:52.520 align:start position:0%
<00:12:52.510>we're <00:12:52.511>in <00:12:52.513>GA, <00:12:52.514>Ollie <00:12:52.516>is <00:12:52.517>now <00:12:52.519>quite

00:12:52.520 --> 00:12:54.470 align:start position:0%
<00:12:52.520>we're <00:12:52.647>in <00:12:52.774>GA, <00:12:52.901>Ollie <00:12:53.029>is <00:12:53.156>now <00:12:53.283>quite <00:12:53.410>cost-effective <00:12:53.538>when <00:12:53.665>comes <00:12:53.792>to <00:12:53.920>tokens,

00:12:54.470 --> 00:12:54.480 align:start position:0%
<00:12:54.470>cost-effective <00:12:54.472>when <00:12:54.474>comes <00:12:54.476>to <00:12:54.479>tokens,

00:12:54.480 --> 00:12:57.110 align:start position:0%
<00:12:54.480>cost-effective <00:12:54.596>when <00:12:54.713>comes <00:12:54.830>to <00:12:54.946>tokens, <00:12:55.063>using <00:12:55.180>cash, <00:12:55.296>and <00:12:55.413>so <00:12:55.530>on. <00:12:55.646>Uh <00:12:55.763>but <00:12:55.880>then

00:12:57.110 --> 00:12:57.120 align:start position:0%
<00:12:57.110>using <00:12:57.111>cash, <00:12:57.112>and <00:12:57.113>so <00:12:57.115>on. <00:12:57.116>Uh <00:12:57.117>but <00:12:57.119>then

00:12:57.120 --> 00:12:59.510 align:start position:0%
<00:12:57.120>using <00:12:57.260>cash, <00:12:57.400>and <00:12:57.540>so <00:12:57.680>on. <00:12:57.820>Uh <00:12:57.960>but <00:12:58.100>then <00:12:58.240>something <00:12:58.380>else <00:12:58.520>happened.

00:12:59.510 --> 00:12:59.520 align:start position:0%
<00:12:59.510>something <00:12:59.514>else <00:12:59.519>happened.

00:12:59.520 --> 00:13:01.670 align:start position:0%
<00:12:59.520>something <00:12:59.694>else <00:12:59.870>happened. <00:13:00.045>Um <00:13:00.220>customers <00:13:00.395>complained <00:13:00.569>that <00:13:00.745>Ollie <00:13:00.920>is

00:13:01.670 --> 00:13:01.680 align:start position:0%
<00:13:01.670>Um <00:13:01.671>customers <00:13:01.673>complained <00:13:01.675>that <00:13:01.677>Ollie <00:13:01.679>is

00:13:01.680 --> 00:13:04.390 align:start position:0%
<00:13:01.680>Um <00:13:01.913>customers <00:13:02.146>complained <00:13:02.380>that <00:13:02.613>Ollie <00:13:02.846>is <00:13:03.079>slow.

00:13:04.400 --> 00:13:05.870 align:start position:0%
<00:13:04.400>And <00:13:04.589>listen, <00:13:04.778>it <00:13:04.967>could <00:13:05.156>be, <00:13:05.345>okay? <00:13:05.534>It <00:13:05.723>takes

00:13:05.870 --> 00:13:05.880 align:start position:0%
<00:13:05.870>And <00:13:05.871>listen, <00:13:05.872>it <00:13:05.873>could <00:13:05.875>be, <00:13:05.876>okay? <00:13:05.877>It <00:13:05.879>takes

00:13:05.880 --> 00:13:08.030 align:start position:0%
<00:13:05.880>And <00:13:05.996>listen, <00:13:06.113>it <00:13:06.230>could <00:13:06.346>be, <00:13:06.463>okay? <00:13:06.580>It <00:13:06.696>takes <00:13:06.813>time <00:13:06.930>to <00:13:07.046>investigate <00:13:07.163>production <00:13:07.280>issues,

00:13:08.030 --> 00:13:08.040 align:start position:0%
<00:13:08.030>time <00:13:08.032>to <00:13:08.034>investigate <00:13:08.036>production <00:13:08.039>issues,

00:13:08.040 --> 00:13:09.510 align:start position:0%
<00:13:08.040>time <00:13:08.150>to <00:13:08.260>investigate <00:13:08.370>production <00:13:08.481>issues, <00:13:08.591>right? <00:13:08.701>Like, <00:13:08.811>how <00:13:08.922>long <00:13:09.032>would <00:13:09.142>it <00:13:09.252>take <00:13:09.363>for

00:13:09.510 --> 00:13:09.520 align:start position:0%
<00:13:09.510>right? <00:13:09.511>Like, <00:13:09.512>how <00:13:09.513>long <00:13:09.515>would <00:13:09.516>it <00:13:09.517>take <00:13:09.519>for

00:13:09.520 --> 00:13:11.630 align:start position:0%
<00:13:09.520>right? <00:13:09.613>Like, <00:13:09.706>how <00:13:09.800>long <00:13:09.893>would <00:13:09.986>it <00:13:10.079>take <00:13:10.173>for <00:13:10.266>a <00:13:10.360>human <00:13:10.453>being <00:13:10.546>to <00:13:10.640>do <00:13:10.733>that? <00:13:10.826>Sometimes <00:13:10.920>it

00:13:11.630 --> 00:13:11.640 align:start position:0%
<00:13:11.630>a <00:13:11.631>human <00:13:11.632>being <00:13:11.633>to <00:13:11.635>do <00:13:11.636>that? <00:13:11.637>Sometimes <00:13:11.639>it

00:13:11.640 --> 00:13:14.630 align:start position:0%
<00:13:11.640>a <00:13:11.733>human <00:13:11.826>being <00:13:11.920>to <00:13:12.013>do <00:13:12.106>that? <00:13:12.199>Sometimes <00:13:12.293>it <00:13:12.386>could <00:13:12.480>be <00:13:12.573>15 <00:13:12.666>minutes, <00:13:12.760>but <00:13:12.853>often <00:13:12.946>times <00:13:13.040>it

00:13:14.630 --> 00:13:14.640 align:start position:0%
<00:13:14.630>could <00:13:14.631>be <00:13:14.632>15 <00:13:14.633>minutes, <00:13:14.635>but <00:13:14.636>often <00:13:14.637>times <00:13:14.639>it

00:13:14.640 --> 00:13:16.870 align:start position:0%
<00:13:14.640>could <00:13:14.733>be <00:13:14.826>15 <00:13:14.920>minutes, <00:13:15.013>but <00:13:15.106>often <00:13:15.199>times <00:13:15.293>it <00:13:15.386>could <00:13:15.480>be <00:13:15.573>2 <00:13:15.666>hours <00:13:15.760>or <00:13:15.853>5 <00:13:15.946>hours, <00:13:16.040>just

00:13:16.870 --> 00:13:16.880 align:start position:0%
<00:13:16.870>could <00:13:16.871>be <00:13:16.872>2 <00:13:16.873>hours <00:13:16.875>or <00:13:16.876>5 <00:13:16.877>hours, <00:13:16.879>just

00:13:16.880 --> 00:13:19.470 align:start position:0%
<00:13:16.880>could <00:13:17.020>be <00:13:17.160>2 <00:13:17.300>hours <00:13:17.439>or <00:13:17.580>5 <00:13:17.720>hours, <00:13:17.860>just <00:13:18.000>opening <00:13:18.140>logs, <00:13:18.280>filtering,

00:13:19.470 --> 00:13:19.480 align:start position:0%
<00:13:19.470>opening <00:13:19.474>logs, <00:13:19.479>filtering,

00:13:19.480 --> 00:13:21.990 align:start position:0%
<00:13:19.480>opening <00:13:19.680>logs, <00:13:19.880>filtering, <00:13:20.080>writing <00:13:20.280>queries, <00:13:20.480>uh <00:13:20.680>understanding <00:13:20.880>what's

00:13:21.990 --> 00:13:22.000 align:start position:0%
<00:13:21.990>writing <00:13:21.992>queries, <00:13:21.994>uh <00:13:21.996>understanding <00:13:21.999>what's

00:13:22.000 --> 00:13:24.190 align:start position:0%
<00:13:22.000>writing <00:13:22.140>queries, <00:13:22.280>uh <00:13:22.420>understanding <00:13:22.560>what's <00:13:22.700>going <00:13:22.840>on. <00:13:22.980>These <00:13:23.120>things <00:13:23.260>take <00:13:23.400>time.

00:13:24.190 --> 00:13:24.200 align:start position:0%
<00:13:24.190>going <00:13:24.191>on. <00:13:24.193>These <00:13:24.195>things <00:13:24.197>take <00:13:24.199>time.

00:13:24.200 --> 00:13:25.510 align:start position:0%
<00:13:24.200>going <00:13:24.396>on. <00:13:24.593>These <00:13:24.789>things <00:13:24.986>take <00:13:25.182>time. <00:13:25.379>Um

00:13:25.510 --> 00:13:25.520 align:start position:0%
Um

00:13:25.520 --> 00:13:26.750 align:start position:0%
<00:13:25.520>Um <00:13:26.627>so,

00:13:26.750 --> 00:13:26.760 align:start position:0%
so,

00:13:26.760 --> 00:13:28.710 align:start position:0%
<00:13:26.760>so, <00:13:27.040>we <00:13:27.319>decided <00:13:27.600>to <00:13:27.880>check <00:13:28.160>that.

00:13:28.710 --> 00:13:28.720 align:start position:0%
<00:13:28.710>we <00:13:28.712>decided <00:13:28.714>to <00:13:28.716>check <00:13:28.719>that.

00:13:28.720 --> 00:13:31.710 align:start position:0%
<00:13:28.720>we <00:13:28.847>decided <00:13:28.974>to <00:13:29.101>check <00:13:29.229>that. <00:13:29.356>And <00:13:29.483>we <00:13:29.610>benchmark <00:13:29.738>cloud <00:13:29.865>code <00:13:29.992>connected <00:13:30.120>to

00:13:31.710 --> 00:13:31.720 align:start position:0%
<00:13:31.710>And <00:13:31.711>we <00:13:31.713>benchmark <00:13:31.714>cloud <00:13:31.716>code <00:13:31.717>connected <00:13:31.719>to

00:13:31.720 --> 00:13:33.630 align:start position:0%
<00:13:31.720>And <00:13:31.847>we <00:13:31.974>benchmark <00:13:32.101>cloud <00:13:32.229>code <00:13:32.356>connected <00:13:32.483>to <00:13:32.610>our <00:13:32.738>MCP, <00:13:32.865>so <00:13:32.992>it <00:13:33.120>actually

00:13:33.630 --> 00:13:33.640 align:start position:0%
<00:13:33.630>our <00:13:33.632>MCP, <00:13:33.634>so <00:13:33.636>it <00:13:33.639>actually

00:13:33.640 --> 00:13:35.350 align:start position:0%
<00:13:33.640>our <00:13:33.767>MCP, <00:13:33.894>so <00:13:34.021>it <00:13:34.149>actually <00:13:34.276>has <00:13:34.403>access <00:13:34.530>to <00:13:34.658>all <00:13:34.785>the <00:13:34.912>telemetry <00:13:35.040>data

00:13:35.350 --> 00:13:35.360 align:start position:0%
<00:13:35.350>has <00:13:35.351>access <00:13:35.353>to <00:13:35.354>all <00:13:35.356>the <00:13:35.357>telemetry <00:13:35.359>data

00:13:35.360 --> 00:13:37.430 align:start position:0%
<00:13:35.360>has <00:13:35.476>access <00:13:35.593>to <00:13:35.710>all <00:13:35.826>the <00:13:35.943>telemetry <00:13:36.060>data <00:13:36.176>from <00:13:36.293>production, <00:13:36.410>and <00:13:36.526>we <00:13:36.643>compare <00:13:36.760>this

00:13:37.430 --> 00:13:37.440 align:start position:0%
<00:13:37.430>from <00:13:37.431>production, <00:13:37.433>and <00:13:37.435>we <00:13:37.437>compare <00:13:37.439>this

00:13:37.440 --> 00:13:38.950 align:start position:0%
<00:13:37.440>from <00:13:37.634>production, <00:13:37.828>and <00:13:38.022>we <00:13:38.216>compare <00:13:38.410>this <00:13:38.604>with <00:13:38.799>Ollie.

00:13:38.950 --> 00:13:38.960 align:start position:0%
<00:13:38.950>with <00:13:38.959>Ollie.

00:13:38.960 --> 00:13:41.070 align:start position:0%
<00:13:38.960>with <00:13:39.310>Ollie. <00:13:39.660>And <00:13:40.010>guess <00:13:40.360>what?

00:13:41.070 --> 00:13:41.080 align:start position:0%
<00:13:41.070>And <00:13:41.074>guess <00:13:41.079>what?

00:13:41.080 --> 00:13:42.990 align:start position:0%
<00:13:41.080>And <00:13:41.235>guess <00:13:41.391>what? <00:13:41.546>Ollie <00:13:41.702>was <00:13:41.857>not <00:13:42.013>slower. <00:13:42.168>It <00:13:42.324>was <00:13:42.480>actually

00:13:42.990 --> 00:13:43.000 align:start position:0%
<00:13:42.990>Ollie <00:13:42.991>was <00:13:42.993>not <00:13:42.994>slower. <00:13:42.996>It <00:13:42.997>was <00:13:42.999>actually

00:13:43.000 --> 00:13:45.790 align:start position:0%
<00:13:43.000>Ollie <00:13:43.116>was <00:13:43.233>not <00:13:43.350>slower. <00:13:43.466>It <00:13:43.583>was <00:13:43.700>actually <00:13:43.816>faster. <00:13:43.933>It <00:13:44.050>took <00:13:44.166>us <00:13:44.283>less <00:13:44.400>time.

00:13:45.790 --> 00:13:45.800 align:start position:0%
<00:13:45.790>faster. <00:13:45.791>It <00:13:45.793>took <00:13:45.795>us <00:13:45.797>less <00:13:45.799>time.

00:13:45.800 --> 00:13:48.310 align:start position:0%
<00:13:45.800>faster. <00:13:45.939>It <00:13:46.079>took <00:13:46.219>us <00:13:46.359>less <00:13:46.500>time. <00:13:46.640>And <00:13:46.780>the <00:13:46.920>quality <00:13:47.060>was <00:13:47.199>better.

00:13:48.310 --> 00:13:48.320 align:start position:0%
<00:13:48.310>And <00:13:48.312>the <00:13:48.314>quality <00:13:48.316>was <00:13:48.319>better.

00:13:48.320 --> 00:13:49.350 align:start position:0%
<00:13:48.320>And <00:13:48.412>the <00:13:48.505>quality <00:13:48.598>was <00:13:48.690>better. <00:13:48.783>So, <00:13:48.876>then <00:13:48.968>there <00:13:49.061>is <00:13:49.154>an <00:13:49.247>interesting

00:13:49.350 --> 00:13:49.360 align:start position:0%
<00:13:49.350>So, <00:13:49.351>then <00:13:49.353>there <00:13:49.355>is <00:13:49.357>an <00:13:49.359>interesting

00:13:49.360 --> 00:13:52.390 align:start position:0%
<00:13:49.360>So, <00:13:49.487>then <00:13:49.614>there <00:13:49.741>is <00:13:49.869>an <00:13:49.996>interesting <00:13:50.123>question <00:13:50.250>of <00:13:50.378>why <00:13:50.505>customers <00:13:50.632>complain <00:13:50.760>time

00:13:52.390 --> 00:13:52.400 align:start position:0%
<00:13:52.390>question <00:13:52.391>of <00:13:52.393>why <00:13:52.395>customers <00:13:52.397>complain <00:13:52.399>time

00:13:52.400 --> 00:13:55.110 align:start position:0%
<00:13:52.400>question <00:13:52.516>of <00:13:52.633>why <00:13:52.750>customers <00:13:52.866>complain <00:13:52.983>time <00:13:53.100>and <00:13:53.216>time <00:13:53.333>again <00:13:53.449>that <00:13:53.566>Ollie <00:13:53.683>is <00:13:53.800>slow

00:13:55.110 --> 00:13:55.120 align:start position:0%
<00:13:55.110>and <00:13:55.111>time <00:13:55.113>again <00:13:55.114>that <00:13:55.116>Ollie <00:13:55.117>is <00:13:55.119>slow

00:13:55.120 --> 00:13:57.110 align:start position:0%
<00:13:55.120>and <00:13:55.236>time <00:13:55.353>again <00:13:55.470>that <00:13:55.586>Ollie <00:13:55.703>is <00:13:55.820>slow <00:13:55.936>if <00:13:56.053>practically, <00:13:56.170>when <00:13:56.286>you're <00:13:56.403>looking <00:13:56.520>on

00:13:57.110 --> 00:13:57.120 align:start position:0%
<00:13:57.110>if <00:13:57.111>practically, <00:13:57.113>when <00:13:57.115>you're <00:13:57.117>looking <00:13:57.119>on

00:13:57.120 --> 00:13:58.230 align:start position:0%
<00:13:57.120>if <00:13:57.210>practically, <00:13:57.301>when <00:13:57.392>you're <00:13:57.483>looking <00:13:57.574>on <00:13:57.664>the <00:13:57.755>numbers, <00:13:57.846>the <00:13:57.937>numbers <00:13:58.028>tell <00:13:58.119>a

00:13:58.230 --> 00:13:58.240 align:start position:0%
<00:13:58.230>the <00:13:58.231>numbers, <00:13:58.233>the <00:13:58.235>numbers <00:13:58.237>tell <00:13:58.239>a

00:13:58.240 --> 00:14:05.070 align:start position:0%
<00:13:58.240>the <00:13:58.440>numbers, <00:13:58.640>the <00:13:58.840>numbers <00:13:59.040>tell <00:13:59.240>a <00:13:59.440>different <00:13:59.640>story.

00:14:05.080 --> 00:14:06.910 align:start position:0%
<00:14:05.080>And <00:14:05.546>hopefully <00:14:06.013>you <00:14:06.480>can

00:14:06.910 --> 00:14:06.920 align:start position:0%
<00:14:06.910>And <00:14:06.913>hopefully <00:14:06.915>you <00:14:06.919>can

00:14:06.920 --> 00:14:09.270 align:start position:0%
<00:14:06.920>And <00:14:07.094>hopefully <00:14:07.270>you <00:14:07.444>can <00:14:07.620>kind <00:14:07.795>of <00:14:07.969>get <00:14:08.145>the <00:14:08.319>feeling

00:14:09.270 --> 00:14:09.280 align:start position:0%
<00:14:09.270>kind <00:14:09.272>of <00:14:09.274>get <00:14:09.276>the <00:14:09.279>feeling

00:14:09.280 --> 00:14:10.910 align:start position:0%
<00:14:09.280>kind <00:14:09.407>of <00:14:09.534>get <00:14:09.661>the <00:14:09.789>feeling <00:14:09.916>of <00:14:10.043>what's <00:14:10.170>happening <00:14:10.298>on <00:14:10.425>the <00:14:10.552>screen <00:14:10.680>behind

00:14:10.910 --> 00:14:10.920 align:start position:0%
<00:14:10.910>of <00:14:10.911>what's <00:14:10.913>happening <00:14:10.914>on <00:14:10.915>the <00:14:10.917>screen <00:14:10.919>behind

00:14:10.920 --> 00:14:11.910 align:start position:0%
<00:14:10.920>of <00:14:11.047>what's <00:14:11.174>happening <00:14:11.301>on <00:14:11.429>the <00:14:11.556>screen <00:14:11.683>behind <00:14:11.810>me.

00:14:11.910 --> 00:14:11.920 align:start position:0%
me.

00:14:11.920 --> 00:14:13.910 align:start position:0%
<00:14:11.920>me. <00:14:12.075>On <00:14:12.231>this <00:14:12.386>side <00:14:12.542>you <00:14:12.697>have <00:14:12.853>the <00:14:13.008>the <00:14:13.164>old <00:14:13.319>user

00:14:13.910 --> 00:14:13.920 align:start position:0%
<00:14:13.910>On <00:14:13.911>this <00:14:13.912>side <00:14:13.913>you <00:14:13.914>have <00:14:13.915>the <00:14:13.916>the <00:14:13.917>old <00:14:13.919>user

00:14:13.920 --> 00:14:16.110 align:start position:0%
<00:14:13.920>On <00:14:14.007>this <00:14:14.094>side <00:14:14.182>you <00:14:14.270>have <00:14:14.357>the <00:14:14.444>the <00:14:14.532>old <00:14:14.620>user <00:14:14.707>experience <00:14:14.795>of <00:14:14.882>Ollie. <00:14:14.969>So, <00:14:15.057>you <00:14:15.145>see <00:14:15.232>it <00:14:15.319>does

00:14:16.110 --> 00:14:16.120 align:start position:0%
<00:14:16.110>experience <00:14:16.111>of <00:14:16.112>Ollie. <00:14:16.113>So, <00:14:16.115>you <00:14:16.116>see <00:14:16.117>it <00:14:16.119>does

00:14:16.120 --> 00:14:18.110 align:start position:0%
<00:14:16.120>experience <00:14:16.220>of <00:14:16.320>Ollie. <00:14:16.420>So, <00:14:16.520>you <00:14:16.620>see <00:14:16.720>it <00:14:16.820>does <00:14:16.920>have <00:14:17.020>streaming <00:14:17.120>support. <00:14:17.220>It <00:14:17.320>does <00:14:17.420>tell <00:14:17.520>you

00:14:18.110 --> 00:14:18.120 align:start position:0%
<00:14:18.110>have <00:14:18.111>streaming <00:14:18.113>support. <00:14:18.114>It <00:14:18.116>does <00:14:18.117>tell <00:14:18.119>you

00:14:18.120 --> 00:14:20.750 align:start position:0%
<00:14:18.120>have <00:14:18.260>streaming <00:14:18.400>support. <00:14:18.540>It <00:14:18.680>does <00:14:18.820>tell <00:14:18.960>you <00:14:19.100>what <00:14:19.240>it <00:14:19.380>is <00:14:19.520>doing.

00:14:20.750 --> 00:14:20.760 align:start position:0%
<00:14:20.750>what <00:14:20.753>it <00:14:20.756>is <00:14:20.759>doing.

00:14:20.760 --> 00:14:22.630 align:start position:0%
<00:14:20.760>what <00:14:20.900>it <00:14:21.040>is <00:14:21.180>doing. <00:14:21.319>But <00:14:21.460>unlike <00:14:21.600>cloud, <00:14:21.740>look <00:14:21.880>on <00:14:22.020>cloud. <00:14:22.160>Cloud

00:14:22.630 --> 00:14:22.640 align:start position:0%
<00:14:22.630>But <00:14:22.631>unlike <00:14:22.633>cloud, <00:14:22.634>look <00:14:22.636>on <00:14:22.637>cloud. <00:14:22.639>Cloud

00:14:22.640 --> 00:14:24.670 align:start position:0%
<00:14:22.640>But <00:14:22.780>unlike <00:14:22.920>cloud, <00:14:23.060>look <00:14:23.199>on <00:14:23.340>cloud. <00:14:23.480>Cloud <00:14:23.620>is <00:14:23.760>constantly <00:14:23.900>doing <00:14:24.040>something.

00:14:24.670 --> 00:14:24.680 align:start position:0%
<00:14:24.670>is <00:14:24.673>constantly <00:14:24.675>doing <00:14:24.679>something.

00:14:24.680 --> 00:14:28.150 align:start position:0%
<00:14:24.680>is <00:14:24.854>constantly <00:14:25.030>doing <00:14:25.204>something. <00:14:25.380>Maybe <00:14:25.555>just <00:14:25.729>writing <00:14:25.905>random <00:14:26.079>keywords.

00:14:28.150 --> 00:14:28.160 align:start position:0%
<00:14:28.150>Maybe <00:14:28.152>just <00:14:28.154>writing <00:14:28.156>random <00:14:28.159>keywords.

00:14:28.160 --> 00:14:30.950 align:start position:0%
<00:14:28.160>Maybe <00:14:28.334>just <00:14:28.510>writing <00:14:28.685>random <00:14:28.860>keywords. <00:14:29.035>Herding. <00:14:29.209>What <00:14:29.385>is <00:14:29.560>herding?

00:14:30.950 --> 00:14:30.960 align:start position:0%
<00:14:30.950>Herding. <00:14:30.953>What <00:14:30.956>is <00:14:30.959>herding?

00:14:30.960 --> 00:14:32.070 align:start position:0%
<00:14:30.960>Herding. <00:14:31.059>What <00:14:31.159>is <00:14:31.259>herding? <00:14:31.359>Right? <00:14:31.459>Then <00:14:31.559>it <00:14:31.659>probably <00:14:31.759>will <00:14:31.859>change <00:14:31.959>it

00:14:32.070 --> 00:14:32.080 align:start position:0%
<00:14:32.070>Right? <00:14:32.071>Then <00:14:32.073>it <00:14:32.074>probably <00:14:32.076>will <00:14:32.077>change <00:14:32.079>it

00:14:32.080 --> 00:14:33.390 align:start position:0%
<00:14:32.080>Right? <00:14:32.187>Then <00:14:32.294>it <00:14:32.401>probably <00:14:32.508>will <00:14:32.615>change <00:14:32.723>it <00:14:32.830>to <00:14:32.937>something <00:14:33.044>else, <00:14:33.151>but <00:14:33.259>there's

00:14:33.390 --> 00:14:33.400 align:start position:0%
<00:14:33.390>to <00:14:33.392>something <00:14:33.394>else, <00:14:33.396>but <00:14:33.399>there's

00:14:33.400 --> 00:14:35.270 align:start position:0%
<00:14:33.400>to <00:14:33.540>something <00:14:33.680>else, <00:14:33.819>but <00:14:33.959>there's <00:14:34.100>constantly <00:14:34.240>things <00:14:34.380>going <00:14:34.520>on <00:14:34.660>on <00:14:34.800>the

00:14:35.270 --> 00:14:35.280 align:start position:0%
<00:14:35.270>constantly <00:14:35.271>things <00:14:35.273>going <00:14:35.275>on <00:14:35.277>on <00:14:35.279>the

00:14:35.280 --> 00:14:37.310 align:start position:0%
<00:14:35.280>constantly <00:14:35.407>things <00:14:35.534>going <00:14:35.661>on <00:14:35.789>on <00:14:35.916>the <00:14:36.043>screen. <00:14:36.170>Okay? <00:14:36.298>So, <00:14:36.425>it's <00:14:36.552>clear <00:14:36.680>that

00:14:37.310 --> 00:14:37.320 align:start position:0%
<00:14:37.310>screen. <00:14:37.311>Okay? <00:14:37.313>So, <00:14:37.315>it's <00:14:37.317>clear <00:14:37.319>that

00:14:37.320 --> 00:14:40.230 align:start position:0%
<00:14:37.320>screen. <00:14:37.436>Okay? <00:14:37.553>So, <00:14:37.670>it's <00:14:37.786>clear <00:14:37.903>that <00:14:38.020>something <00:14:38.136>is <00:14:38.253>going <00:14:38.370>on. <00:14:38.486>Uh <00:14:38.603>whereas <00:14:38.720>in

00:14:40.230 --> 00:14:40.240 align:start position:0%
<00:14:40.230>something <00:14:40.231>is <00:14:40.233>going <00:14:40.234>on. <00:14:40.236>Uh <00:14:40.237>whereas <00:14:40.239>in

00:14:40.240 --> 00:14:42.590 align:start position:0%
<00:14:40.240>something <00:14:40.367>is <00:14:40.494>going <00:14:40.621>on. <00:14:40.749>Uh <00:14:40.876>whereas <00:14:41.003>in <00:14:41.130>Ollie, <00:14:41.258>it's <00:14:41.385>less <00:14:41.512>the <00:14:41.640>case.

00:14:42.590 --> 00:14:42.600 align:start position:0%
<00:14:42.590>Ollie, <00:14:42.592>it's <00:14:42.594>less <00:14:42.596>the <00:14:42.599>case.

00:14:42.600 --> 00:14:44.030 align:start position:0%
<00:14:42.600>Ollie, <00:14:42.728>it's <00:14:42.857>less <00:14:42.986>the <00:14:43.114>case. <00:14:43.243>And <00:14:43.372>that's <00:14:43.500>where <00:14:43.629>where <00:14:43.758>we <00:14:43.887>realized,

00:14:44.030 --> 00:14:44.040 align:start position:0%
<00:14:44.030>And <00:14:44.031>that's <00:14:44.033>where <00:14:44.035>where <00:14:44.037>we <00:14:44.039>realized,

00:14:44.040 --> 00:14:47.590 align:start position:0%
<00:14:44.040>And <00:14:44.156>that's <00:14:44.273>where <00:14:44.390>where <00:14:44.506>we <00:14:44.623>realized, <00:14:44.740>"Hey, <00:14:44.856>listen. <00:14:44.973>Um <00:14:45.089>actually <00:14:45.206>there <00:14:45.323>could <00:14:45.439>be

00:14:47.590 --> 00:14:47.600 align:start position:0%
<00:14:47.590>"Hey, <00:14:47.591>listen. <00:14:47.593>Um <00:14:47.594>actually <00:14:47.596>there <00:14:47.597>could <00:14:47.599>be

00:14:47.600 --> 00:14:50.590 align:start position:0%
<00:14:47.600>"Hey, <00:14:47.700>listen. <00:14:47.800>Um <00:14:47.900>actually <00:14:48.000>there <00:14:48.100>could <00:14:48.200>be <00:14:48.300>maybe <00:14:48.400>maybe <00:14:48.500>just <00:14:48.600>maybe <00:14:48.700>um <00:14:48.800>Ollie <00:14:48.900>is <00:14:49.000>not

00:14:50.590 --> 00:14:50.600 align:start position:0%
<00:14:50.590>maybe <00:14:50.591>maybe <00:14:50.592>just <00:14:50.593>maybe <00:14:50.595>um <00:14:50.596>Ollie <00:14:50.597>is <00:14:50.599>not

00:14:50.600 --> 00:14:54.510 align:start position:0%
<00:14:50.600>maybe <00:14:50.700>maybe <00:14:50.800>just <00:14:50.900>maybe <00:14:51.000>um <00:14:51.100>Ollie <00:14:51.200>is <00:14:51.300>not <00:14:51.400>really <00:14:51.500>slower, <00:14:51.600>but <00:14:51.700>it <00:14:51.800>just <00:14:51.900>feels <00:14:52.000>slower.

00:14:54.510 --> 00:14:54.520 align:start position:0%
<00:14:54.510>really <00:14:54.511>slower, <00:14:54.513>but <00:14:54.514>it <00:14:54.516>just <00:14:54.517>feels <00:14:54.519>slower.

00:14:54.520 --> 00:14:57.550 align:start position:0%
<00:14:54.520>really <00:14:54.636>slower, <00:14:54.753>but <00:14:54.870>it <00:14:54.986>just <00:14:55.103>feels <00:14:55.220>slower. <00:14:55.336>It's <00:14:55.453>about <00:14:55.569>feeling, <00:14:55.686>not <00:14:55.803>about <00:14:55.920>numbers.

00:14:57.550 --> 00:14:57.560 align:start position:0%
<00:14:57.550>It's <00:14:57.551>about <00:14:57.553>feeling, <00:14:57.555>not <00:14:57.557>about <00:14:57.559>numbers.

00:14:57.560 --> 00:14:58.870 align:start position:0%
<00:14:57.560>It's <00:14:57.658>about <00:14:57.756>feeling, <00:14:57.854>not <00:14:57.953>about <00:14:58.051>numbers. <00:14:58.149>Because <00:14:58.247>the <00:14:58.346>numbers <00:14:58.444>tell <00:14:58.542>us <00:14:58.640>tell <00:14:58.739>a

00:14:58.870 --> 00:14:58.880 align:start position:0%
<00:14:58.870>Because <00:14:58.871>the <00:14:58.873>numbers <00:14:58.874>tell <00:14:58.876>us <00:14:58.877>tell <00:14:58.879>a

00:14:58.880 --> 00:15:00.430 align:start position:0%
<00:14:58.880>Because <00:14:59.054>the <00:14:59.228>numbers <00:14:59.403>tell <00:14:59.577>us <00:14:59.751>tell <00:14:59.926>a <00:15:00.100>different <00:15:00.275>story.

00:15:00.430 --> 00:15:00.440 align:start position:0%
<00:15:00.430>different <00:15:00.439>story.

00:15:00.440 --> 00:15:02.190 align:start position:0%
<00:15:00.440>different <00:15:01.140>story. <00:15:01.840>So,

00:15:02.190 --> 00:15:02.200 align:start position:0%
So,

00:15:02.200 --> 00:15:04.430 align:start position:0%
<00:15:02.200>So, <00:15:02.375>we <00:15:02.550>changed <00:15:02.725>that. <00:15:02.900>We <00:15:03.075>make <00:15:03.250>Ollie <00:15:03.425>think <00:15:03.600>out

00:15:04.430 --> 00:15:04.440 align:start position:0%
<00:15:04.430>we <00:15:04.431>changed <00:15:04.432>that. <00:15:04.433>We <00:15:04.435>make <00:15:04.436>Ollie <00:15:04.437>think <00:15:04.439>out

00:15:04.440 --> 00:15:07.710 align:start position:0%
<00:15:04.440>we <00:15:04.556>changed <00:15:04.673>that. <00:15:04.790>We <00:15:04.906>make <00:15:05.023>Ollie <00:15:05.140>think <00:15:05.256>out <00:15:05.373>loud. <00:15:05.490>Reasoning, <00:15:05.606>thinking, <00:15:05.723>tool <00:15:05.840>calling.

00:15:07.710 --> 00:15:07.720 align:start position:0%
<00:15:07.710>loud. <00:15:07.712>Reasoning, <00:15:07.714>thinking, <00:15:07.716>tool <00:15:07.719>calling.

00:15:07.720 --> 00:15:09.190 align:start position:0%
<00:15:07.720>loud. <00:15:07.984>Reasoning, <00:15:08.249>thinking, <00:15:08.513>tool <00:15:08.778>calling. <00:15:09.043>Everything

00:15:09.190 --> 00:15:09.200 align:start position:0%
Everything

00:15:09.200 --> 00:15:11.270 align:start position:0%
<00:15:09.200>Everything <00:15:10.600>became

00:15:11.270 --> 00:15:11.280 align:start position:0%
became

00:15:11.280 --> 00:15:13.350 align:start position:0%
<00:15:11.280>became <00:15:11.746>everything <00:15:12.213>became <00:15:12.680>verbose.

00:15:13.350 --> 00:15:13.360 align:start position:0%
<00:15:13.350>everything <00:15:13.354>became <00:15:13.359>verbose.

00:15:13.360 --> 00:15:15.270 align:start position:0%
<00:15:13.360>everything <00:15:13.535>became <00:15:13.710>verbose. <00:15:13.885>Everything <00:15:14.060>became <00:15:14.235>clear <00:15:14.410>and <00:15:14.585>onto <00:15:14.760>the

00:15:15.270 --> 00:15:15.280 align:start position:0%
<00:15:15.270>Everything <00:15:15.271>became <00:15:15.273>clear <00:15:15.275>and <00:15:15.277>onto <00:15:15.279>the

00:15:15.280 --> 00:15:17.190 align:start position:0%
<00:15:15.280>Everything <00:15:15.480>became <00:15:15.680>clear <00:15:15.880>and <00:15:16.079>onto <00:15:16.280>the <00:15:16.480>screen. <00:15:16.680>Um

00:15:17.190 --> 00:15:17.200 align:start position:0%
<00:15:17.190>screen. <00:15:17.199>Um

00:15:17.200 --> 00:15:20.230 align:start position:0%
<00:15:17.200>screen. <00:15:17.375>Um <00:15:17.550>what <00:15:17.725>we're <00:15:17.900>seeing <00:15:18.075>now <00:15:18.250>is <00:15:18.425>how <00:15:18.600>Ollie

00:15:20.230 --> 00:15:20.240 align:start position:0%
<00:15:20.230>what <00:15:20.231>we're <00:15:20.233>seeing <00:15:20.234>now <00:15:20.236>is <00:15:20.237>how <00:15:20.239>Ollie

00:15:20.240 --> 00:15:23.430 align:start position:0%
<00:15:20.240>what <00:15:20.340>we're <00:15:20.440>seeing <00:15:20.540>now <00:15:20.640>is <00:15:20.740>how <00:15:20.840>Ollie <00:15:20.940>actually <00:15:21.040>sh- <00:15:21.140>what <00:15:21.240>Ollie <00:15:21.340>um <00:15:21.440>is <00:15:21.540>doing <00:15:21.640>in

00:15:23.430 --> 00:15:23.440 align:start position:0%
<00:15:23.430>actually <00:15:23.431>sh- <00:15:23.432>what <00:15:23.433>Ollie <00:15:23.435>um <00:15:23.436>is <00:15:23.437>doing <00:15:23.439>in

00:15:23.440 --> 00:15:25.390 align:start position:0%
<00:15:23.440>actually <00:15:23.567>sh- <00:15:23.694>what <00:15:23.821>Ollie <00:15:23.949>um <00:15:24.076>is <00:15:24.203>doing <00:15:24.330>in <00:15:24.458>terms <00:15:24.585>of <00:15:24.712>user <00:15:24.840>experience

00:15:25.390 --> 00:15:25.400 align:start position:0%
<00:15:25.390>terms <00:15:25.393>of <00:15:25.396>user <00:15:25.399>experience

00:15:25.400 --> 00:15:29.750 align:start position:0%
<00:15:25.400>terms <00:15:25.600>of <00:15:25.800>user <00:15:26.000>experience <00:15:26.199>in <00:15:26.400>the <00:15:26.600>new <00:15:26.800>way.

00:15:29.760 --> 00:15:32.230 align:start position:0%
<00:15:29.760>And <00:15:30.040>what <00:15:30.319>you <00:15:30.600>will <00:15:30.880>see <00:15:31.160>is

00:15:32.230 --> 00:15:32.240 align:start position:0%
<00:15:32.230>And <00:15:32.231>what <00:15:32.233>you <00:15:32.235>will <00:15:32.237>see <00:15:32.239>is

00:15:32.240 --> 00:15:33.790 align:start position:0%
<00:15:32.240>And <00:15:32.356>what <00:15:32.472>you <00:15:32.588>will <00:15:32.705>see <00:15:32.821>is <00:15:32.937>again <00:15:33.053>I <00:15:33.170>think <00:15:33.286>in <00:15:33.402>the <00:15:33.518>traditional <00:15:33.635>web

00:15:33.790 --> 00:15:33.800 align:start position:0%
<00:15:33.790>again <00:15:33.791>I <00:15:33.793>think <00:15:33.794>in <00:15:33.795>the <00:15:33.797>traditional <00:15:33.799>web

00:15:33.800 --> 00:15:36.310 align:start position:0%
<00:15:33.800>again <00:15:33.916>I <00:15:34.033>think <00:15:34.150>in <00:15:34.266>the <00:15:34.383>traditional <00:15:34.500>web <00:15:34.616>application <00:15:34.733>space <00:15:34.849>we <00:15:34.966>had <00:15:35.083>spinners, <00:15:35.199>we

00:15:36.310 --> 00:15:36.320 align:start position:0%
<00:15:36.310>application <00:15:36.311>space <00:15:36.313>we <00:15:36.315>had <00:15:36.317>spinners, <00:15:36.319>we

00:15:36.320 --> 00:15:38.870 align:start position:0%
<00:15:36.320>application <00:15:36.436>space <00:15:36.553>we <00:15:36.670>had <00:15:36.786>spinners, <00:15:36.903>we <00:15:37.020>had <00:15:37.136>skeleton <00:15:37.253>loaders, <00:15:37.370>right <00:15:37.486>we <00:15:37.603>have <00:15:37.720>all

00:15:38.870 --> 00:15:38.880 align:start position:0%
<00:15:38.870>had <00:15:38.871>skeleton <00:15:38.873>loaders, <00:15:38.874>right <00:15:38.876>we <00:15:38.877>have <00:15:38.879>all

00:15:38.880 --> 00:15:41.590 align:start position:0%
<00:15:38.880>had <00:15:39.020>skeleton <00:15:39.160>loaders, <00:15:39.300>right <00:15:39.439>we <00:15:39.580>have <00:15:39.720>all <00:15:39.860>these <00:15:40.000>techniques <00:15:40.140>and <00:15:40.280>tools

00:15:41.590 --> 00:15:41.600 align:start position:0%
<00:15:41.590>these <00:15:41.593>techniques <00:15:41.596>and <00:15:41.599>tools

00:15:41.600 --> 00:15:43.310 align:start position:0%
<00:15:41.600>these <00:15:41.880>techniques <00:15:42.160>and <00:15:42.440>tools <00:15:42.720>to <00:15:43.000>actually

00:15:43.310 --> 00:15:43.320 align:start position:0%
<00:15:43.310>to <00:15:43.319>actually

00:15:43.320 --> 00:15:45.710 align:start position:0%
<00:15:43.320>to <00:15:43.495>actually <00:15:43.670>provide <00:15:43.845>the <00:15:44.020>users <00:15:44.195>with <00:15:44.370>the <00:15:44.545>feeling <00:15:44.720>that

00:15:45.710 --> 00:15:45.720 align:start position:0%
<00:15:45.710>provide <00:15:45.711>the <00:15:45.713>users <00:15:45.714>with <00:15:45.716>the <00:15:45.717>feeling <00:15:45.719>that

00:15:45.720 --> 00:15:47.470 align:start position:0%
<00:15:45.720>provide <00:15:45.836>the <00:15:45.953>users <00:15:46.070>with <00:15:46.186>the <00:15:46.303>feeling <00:15:46.420>that <00:15:46.536>something <00:15:46.653>is <00:15:46.770>going <00:15:46.886>on. <00:15:47.003>Don't <00:15:47.120>worry,

00:15:47.470 --> 00:15:47.480 align:start position:0%
<00:15:47.470>something <00:15:47.471>is <00:15:47.473>going <00:15:47.475>on. <00:15:47.477>Don't <00:15:47.479>worry,

00:15:47.480 --> 00:15:50.230 align:start position:0%
<00:15:47.480>something <00:15:47.580>is <00:15:47.680>going <00:15:47.780>on. <00:15:47.880>Don't <00:15:47.980>worry, <00:15:48.080>we're <00:15:48.180>on <00:15:48.280>it. <00:15:48.380>A <00:15:48.480>process <00:15:48.580>is <00:15:48.680>going <00:15:48.780>on. <00:15:48.880>Um

00:15:50.230 --> 00:15:50.240 align:start position:0%
<00:15:50.230>we're <00:15:50.231>on <00:15:50.232>it. <00:15:50.233>A <00:15:50.234>process <00:15:50.235>is <00:15:50.236>going <00:15:50.237>on. <00:15:50.239>Um

00:15:50.240 --> 00:15:52.430 align:start position:0%
<00:15:50.240>we're <00:15:50.322>on <00:15:50.404>it. <00:15:50.487>A <00:15:50.569>process <00:15:50.651>is <00:15:50.734>going <00:15:50.816>on. <00:15:50.898>Um <00:15:50.981>and <00:15:51.063>as <00:15:51.145>you <00:15:51.228>can <00:15:51.310>see <00:15:51.392>right <00:15:51.475>now <00:15:51.557>Ollie <00:15:51.640>is

00:15:52.430 --> 00:15:52.440 align:start position:0%
<00:15:52.430>and <00:15:52.431>as <00:15:52.432>you <00:15:52.433>can <00:15:52.434>see <00:15:52.435>right <00:15:52.436>now <00:15:52.437>Ollie <00:15:52.439>is

00:15:52.440 --> 00:15:54.630 align:start position:0%
<00:15:52.440>and <00:15:52.547>as <00:15:52.655>you <00:15:52.763>can <00:15:52.870>see <00:15:52.978>right <00:15:53.086>now <00:15:53.193>Ollie <00:15:53.301>is <00:15:53.409>constantly <00:15:53.516>putting <00:15:53.624>something <00:15:53.732>on <00:15:53.840>the

00:15:54.630 --> 00:15:54.640 align:start position:0%
<00:15:54.630>constantly <00:15:54.632>putting <00:15:54.634>something <00:15:54.636>on <00:15:54.639>the

00:15:54.640 --> 00:15:56.990 align:start position:0%
<00:15:54.640>constantly <00:15:54.795>putting <00:15:54.951>something <00:15:55.106>on <00:15:55.262>the <00:15:55.417>screen. <00:15:55.573>Constantly <00:15:55.728>something <00:15:55.884>is <00:15:56.040>going

00:15:56.990 --> 00:15:57.000 align:start position:0%
<00:15:56.990>screen. <00:15:56.992>Constantly <00:15:56.994>something <00:15:56.996>is <00:15:56.999>going

00:15:57.000 --> 00:15:58.870 align:start position:0%
<00:15:57.000>screen. <00:15:57.127>Constantly <00:15:57.254>something <00:15:57.381>is <00:15:57.509>going <00:15:57.636>on. <00:15:57.763>So <00:15:57.890>it's <00:15:58.018>not <00:15:58.145>hidden <00:15:58.272>anymore. <00:15:58.400>You

00:15:58.870 --> 00:15:58.880 align:start position:0%
<00:15:58.870>on. <00:15:58.871>So <00:15:58.873>it's <00:15:58.874>not <00:15:58.876>hidden <00:15:58.877>anymore. <00:15:58.879>You

00:15:58.880 --> 00:16:00.310 align:start position:0%
<00:15:58.880>on. <00:15:59.023>So <00:15:59.165>it's <00:15:59.309>not <00:15:59.452>hidden <00:15:59.595>anymore. <00:15:59.738>You <00:15:59.881>don't <00:16:00.024>see <00:16:00.166>like

00:16:00.310 --> 00:16:00.320 align:start position:0%
<00:16:00.310>don't <00:16:00.314>see <00:16:00.319>like

00:16:00.320 --> 00:16:02.150 align:start position:0%
<00:16:00.320>don't <00:16:00.520>see <00:16:00.720>like <00:16:00.920>assessing <00:16:01.120>service <00:16:01.320>activity <00:16:01.520>on <00:16:01.720>your

00:16:02.150 --> 00:16:02.160 align:start position:0%
<00:16:02.150>assessing <00:16:02.152>service <00:16:02.154>activity <00:16:02.156>on <00:16:02.159>your

00:16:02.160 --> 00:16:03.910 align:start position:0%
<00:16:02.160>assessing <00:16:02.300>service <00:16:02.439>activity <00:16:02.579>on <00:16:02.719>your <00:16:02.860>screen <00:16:03.000>for <00:16:03.140>3 <00:16:03.280>3 <00:16:03.420>minutes <00:16:03.560>without

00:16:03.910 --> 00:16:03.920 align:start position:0%
<00:16:03.910>screen <00:16:03.911>for <00:16:03.913>3 <00:16:03.915>3 <00:16:03.917>minutes <00:16:03.919>without

00:16:03.920 --> 00:16:07.310 align:start position:0%
<00:16:03.920>screen <00:16:04.075>for <00:16:04.231>3 <00:16:04.386>3 <00:16:04.542>minutes <00:16:04.697>without <00:16:04.853>understanding <00:16:05.008>what's <00:16:05.164>going <00:16:05.319>on.

00:16:07.320 --> 00:16:08.670 align:start position:0%
<00:16:07.320>So <00:16:08.535>um

00:16:08.670 --> 00:16:08.680 align:start position:0%
<00:16:08.670>So <00:16:08.679>um

00:16:08.680 --> 00:16:10.190 align:start position:0%
<00:16:08.680>So <00:16:09.019>um <00:16:09.359>that's <00:16:09.699>another <00:16:10.039>learning.

00:16:10.190 --> 00:16:10.200 align:start position:0%
<00:16:10.190>that's <00:16:10.194>another <00:16:10.199>learning.

00:16:10.200 --> 00:16:11.830 align:start position:0%
<00:16:10.200>that's <00:16:10.480>another <00:16:10.760>learning. <00:16:11.040>Um <00:16:11.320>we <00:16:11.600>had

00:16:11.830 --> 00:16:11.840 align:start position:0%
<00:16:11.830>Um <00:16:11.834>we <00:16:11.839>had

00:16:11.840 --> 00:16:14.230 align:start position:0%
<00:16:11.840>Um <00:16:12.120>we <00:16:12.400>had <00:16:12.680>0 <00:16:12.960>millisecond <00:16:13.240>faster

00:16:14.230 --> 00:16:14.240 align:start position:0%
<00:16:14.230>0 <00:16:14.234>millisecond <00:16:14.239>faster

00:16:14.240 --> 00:16:15.710 align:start position:0%
<00:16:14.240>0 <00:16:14.681>millisecond <00:16:15.122>faster <00:16:15.563>agent

00:16:15.710 --> 00:16:15.720 align:start position:0%
agent

00:16:15.720 --> 00:16:18.310 align:start position:0%
<00:16:15.720>agent <00:16:16.070>that <00:16:16.420>felt <00:16:16.770>dramatically <00:16:17.120>faster.

00:16:18.310 --> 00:16:18.320 align:start position:0%
<00:16:18.310>that <00:16:18.313>felt <00:16:18.316>dramatically <00:16:18.319>faster.

00:16:18.320 --> 00:16:20.510 align:start position:0%
<00:16:18.320>that <00:16:18.447>felt <00:16:18.574>dramatically <00:16:18.701>faster. <00:16:18.829>Like <00:16:18.956>I <00:16:19.083>I'm <00:16:19.210>telling <00:16:19.338>you <00:16:19.465>we <00:16:19.592>haven't <00:16:19.720>got

00:16:20.510 --> 00:16:20.520 align:start position:0%
<00:16:20.510>Like <00:16:20.511>I <00:16:20.512>I'm <00:16:20.513>telling <00:16:20.515>you <00:16:20.516>we <00:16:20.517>haven't <00:16:20.519>got

00:16:20.520 --> 00:16:22.350 align:start position:0%
<00:16:20.520>Like <00:16:20.627>I <00:16:20.735>I'm <00:16:20.843>telling <00:16:20.950>you <00:16:21.058>we <00:16:21.166>haven't <00:16:21.273>got <00:16:21.381>any <00:16:21.489>complaints <00:16:21.596>since <00:16:21.704>then <00:16:21.812>about <00:16:21.920>Ollie

00:16:22.350 --> 00:16:22.360 align:start position:0%
<00:16:22.350>any <00:16:22.351>complaints <00:16:22.353>since <00:16:22.355>then <00:16:22.357>about <00:16:22.359>Ollie

00:16:22.360 --> 00:16:23.710 align:start position:0%
<00:16:22.360>any <00:16:22.533>complaints <00:16:22.707>since <00:16:22.880>then <00:16:23.054>about <00:16:23.227>Ollie <00:16:23.401>being <00:16:23.575>slow.

00:16:23.710 --> 00:16:23.720 align:start position:0%
<00:16:23.710>being <00:16:23.719>slow.

00:16:23.720 --> 00:16:25.950 align:start position:0%
<00:16:23.720>being <00:16:23.920>slow. <00:16:24.120>Um <00:16:24.320>actually <00:16:24.520>you <00:16:24.720>know <00:16:24.920>something <00:16:25.120>funny

00:16:25.950 --> 00:16:25.960 align:start position:0%
<00:16:25.950>Um <00:16:25.951>actually <00:16:25.953>you <00:16:25.955>know <00:16:25.957>something <00:16:25.959>funny

00:16:25.960 --> 00:16:28.390 align:start position:0%
<00:16:25.960>Um <00:16:26.067>actually <00:16:26.175>you <00:16:26.283>know <00:16:26.390>something <00:16:26.498>funny <00:16:26.606>the <00:16:26.713>feeling <00:16:26.821>we <00:16:26.929>got <00:16:27.036>now <00:16:27.144>from <00:16:27.252>customers <00:16:27.360>is

00:16:28.390 --> 00:16:28.400 align:start position:0%
<00:16:28.390>the <00:16:28.391>feeling <00:16:28.392>we <00:16:28.393>got <00:16:28.395>now <00:16:28.396>from <00:16:28.397>customers <00:16:28.399>is

00:16:28.400 --> 00:16:30.110 align:start position:0%
<00:16:28.400>the <00:16:28.516>feeling <00:16:28.633>we <00:16:28.750>got <00:16:28.866>now <00:16:28.983>from <00:16:29.100>customers <00:16:29.216>is <00:16:29.333>actually <00:16:29.449>sometimes <00:16:29.566>the <00:16:29.683>exact <00:16:29.800>opposite.

00:16:30.110 --> 00:16:30.120 align:start position:0%
<00:16:30.110>actually <00:16:30.112>sometimes <00:16:30.114>the <00:16:30.116>exact <00:16:30.119>opposite.

00:16:30.120 --> 00:16:31.510 align:start position:0%
<00:16:30.120>actually <00:16:30.233>sometimes <00:16:30.347>the <00:16:30.461>exact <00:16:30.574>opposite. <00:16:30.688>Sometimes <00:16:30.802>they <00:16:30.916>tell <00:16:31.029>us <00:16:31.143>that <00:16:31.257>they <00:16:31.371>feel

00:16:31.510 --> 00:16:31.520 align:start position:0%
<00:16:31.510>Sometimes <00:16:31.511>they <00:16:31.513>tell <00:16:31.514>us <00:16:31.516>that <00:16:31.517>they <00:16:31.519>feel

00:16:31.520 --> 00:16:33.710 align:start position:0%
<00:16:31.520>Sometimes <00:16:31.647>they <00:16:31.774>tell <00:16:31.901>us <00:16:32.029>that <00:16:32.156>they <00:16:32.283>feel <00:16:32.410>that <00:16:32.538>Ollie <00:16:32.665>is <00:16:32.792>too <00:16:32.920>fast

00:16:33.710 --> 00:16:33.720 align:start position:0%
<00:16:33.710>that <00:16:33.712>Ollie <00:16:33.714>is <00:16:33.716>too <00:16:33.719>fast

00:16:33.720 --> 00:16:35.350 align:start position:0%
<00:16:33.720>that <00:16:33.847>Ollie <00:16:33.974>is <00:16:34.101>too <00:16:34.229>fast <00:16:34.356>and <00:16:34.483>they <00:16:34.610>feel <00:16:34.738>that <00:16:34.865>it's <00:16:34.992>not <00:16:35.120>thinking

00:16:35.350 --> 00:16:35.360 align:start position:0%
<00:16:35.350>and <00:16:35.351>they <00:16:35.353>feel <00:16:35.354>that <00:16:35.356>it's <00:16:35.357>not <00:16:35.359>thinking

00:16:35.360 --> 00:16:36.750 align:start position:0%
<00:16:35.360>and <00:16:35.538>they <00:16:35.717>feel <00:16:35.896>that <00:16:36.074>it's <00:16:36.253>not <00:16:36.432>thinking <00:16:36.611>enough.

00:16:36.750 --> 00:16:36.760 align:start position:0%
enough.

00:16:36.760 --> 00:16:38.150 align:start position:0%
<00:16:36.760>enough. <00:16:36.938>Okay, <00:16:37.117>so <00:16:37.296>it's <00:16:37.474>really <00:16:37.653>I <00:16:37.832>think <00:16:38.011>an

00:16:38.150 --> 00:16:38.160 align:start position:0%
<00:16:38.150>Okay, <00:16:38.151>so <00:16:38.153>it's <00:16:38.154>really <00:16:38.156>I <00:16:38.157>think <00:16:38.159>an

00:16:38.160 --> 00:16:40.841 align:start position:0%
<00:16:38.160>Okay, <00:16:38.300>so <00:16:38.439>it's <00:16:38.579>really <00:16:38.719>I <00:16:38.860>think <00:16:39.000>an <00:16:39.140>interesting <00:16:39.280>kind <00:16:39.420>of <00:16:39.560>balance

00:16:40.841 --> 00:16:40.851 align:start position:0%
<00:16:40.841>interesting <00:16:40.844>kind <00:16:40.847>of <00:16:40.850>balance

00:16:40.851 --> 00:16:40.990 align:start position:0%
<00:16:40.851>interesting <00:16:40.876>kind <00:16:40.901>of <00:16:40.926>balance <00:16:40.951>&gt;&gt; <00:16:40.976>[snorts]

00:16:40.990 --> 00:16:41.000 align:start position:0%
<00:16:40.990>&gt;&gt; <00:16:40.999>[snorts]

00:16:41.000 --> 00:16:42.590 align:start position:0%
<00:16:41.000>&gt;&gt; <00:16:41.140>[snorts] <00:16:41.280>&gt;&gt; <00:16:41.420>of <00:16:41.560>how <00:16:41.700>do <00:16:41.840>we <00:16:41.980>find <00:16:42.120>this <00:16:42.260>balance <00:16:42.400>of

00:16:42.590 --> 00:16:42.600 align:start position:0%
<00:16:42.590>&gt;&gt; <00:16:42.591>of <00:16:42.592>how <00:16:42.593>do <00:16:42.594>we <00:16:42.595>find <00:16:42.596>this <00:16:42.597>balance <00:16:42.599>of

00:16:42.600 --> 00:16:45.190 align:start position:0%
<00:16:42.600>&gt;&gt; <00:16:42.700>of <00:16:42.800>how <00:16:42.900>do <00:16:43.000>we <00:16:43.100>find <00:16:43.200>this <00:16:43.300>balance <00:16:43.400>of <00:16:43.500>creating <00:16:43.600>this <00:16:43.700>experience <00:16:43.800>of <00:16:43.900>you <00:16:44.000>can

00:16:45.190 --> 00:16:45.200 align:start position:0%
<00:16:45.190>creating <00:16:45.191>this <00:16:45.193>experience <00:16:45.195>of <00:16:45.197>you <00:16:45.199>can

00:16:45.200 --> 00:16:47.030 align:start position:0%
<00:16:45.200>creating <00:16:45.355>this <00:16:45.511>experience <00:16:45.666>of <00:16:45.822>you <00:16:45.977>can <00:16:46.133>really <00:16:46.288>on <00:16:46.444>one <00:16:46.600>hand

00:16:47.030 --> 00:16:47.040 align:start position:0%
<00:16:47.030>really <00:16:47.033>on <00:16:47.036>one <00:16:47.039>hand

00:16:47.040 --> 00:16:49.750 align:start position:0%
<00:16:47.040>really <00:16:47.167>on <00:16:47.294>one <00:16:47.421>hand <00:16:47.549>don't <00:16:47.676>worry <00:16:47.803>the <00:16:47.930>agent <00:16:48.058>is <00:16:48.185>working <00:16:48.312>for <00:16:48.439>you

00:16:49.750 --> 00:16:49.760 align:start position:0%
<00:16:49.750>don't <00:16:49.751>worry <00:16:49.752>the <00:16:49.753>agent <00:16:49.755>is <00:16:49.756>working <00:16:49.757>for <00:16:49.759>you

00:16:49.760 --> 00:16:52.070 align:start position:0%
<00:16:49.760>don't <00:16:49.853>worry <00:16:49.946>the <00:16:50.040>agent <00:16:50.133>is <00:16:50.226>working <00:16:50.319>for <00:16:50.413>you <00:16:50.506>on <00:16:50.600>the <00:16:50.693>other <00:16:50.786>hand <00:16:50.880>it's <00:16:50.973>really <00:16:51.066>doing <00:16:51.160>a

00:16:52.070 --> 00:16:52.080 align:start position:0%
<00:16:52.070>on <00:16:52.071>the <00:16:52.072>other <00:16:52.073>hand <00:16:52.075>it's <00:16:52.076>really <00:16:52.077>doing <00:16:52.079>a

00:16:52.080 --> 00:16:53.750 align:start position:0%
<00:16:52.080>on <00:16:52.207>the <00:16:52.334>other <00:16:52.461>hand <00:16:52.589>it's <00:16:52.716>really <00:16:52.843>doing <00:16:52.970>a <00:16:53.098>work <00:16:53.225>you <00:16:53.352>can <00:16:53.480>trust.

00:16:53.750 --> 00:16:53.760 align:start position:0%
<00:16:53.750>work <00:16:53.753>you <00:16:53.756>can <00:16:53.759>trust.

00:16:53.760 --> 00:16:55.110 align:start position:0%
<00:16:53.760>work <00:16:53.895>you <00:16:54.030>can <00:16:54.165>trust. <00:16:54.300>Right <00:16:54.435>because <00:16:54.570>the <00:16:54.705>concern <00:16:54.840>with <00:16:54.975>Ollie

00:16:55.110 --> 00:16:55.120 align:start position:0%
<00:16:55.110>Right <00:16:55.111>because <00:16:55.113>the <00:16:55.115>concern <00:16:55.117>with <00:16:55.119>Ollie

00:16:55.120 --> 00:16:58.150 align:start position:0%
<00:16:55.120>Right <00:16:55.227>because <00:16:55.335>the <00:16:55.443>concern <00:16:55.550>with <00:16:55.658>Ollie <00:16:55.766>being <00:16:55.873>too <00:16:55.981>fast <00:16:56.089>is <00:16:56.196>not <00:16:56.304>that <00:16:56.412>someone <00:16:56.520>is

00:16:58.150 --> 00:16:58.160 align:start position:0%
<00:16:58.150>being <00:16:58.151>too <00:16:58.152>fast <00:16:58.153>is <00:16:58.155>not <00:16:58.156>that <00:16:58.157>someone <00:16:58.159>is

00:16:58.160 --> 00:17:00.910 align:start position:0%
<00:16:58.160>being <00:16:58.276>too <00:16:58.393>fast <00:16:58.510>is <00:16:58.626>not <00:16:58.743>that <00:16:58.860>someone <00:16:58.976>is <00:16:59.093>mad <00:16:59.209>at <00:16:59.326>getting <00:16:59.443>fast <00:16:59.560>results.

00:17:00.910 --> 00:17:00.920 align:start position:0%
<00:17:00.910>mad <00:17:00.912>at <00:17:00.914>getting <00:17:00.916>fast <00:17:00.919>results.

00:17:00.920 --> 00:17:02.070 align:start position:0%
<00:17:00.920>mad <00:17:01.092>at <00:17:01.265>getting <00:17:01.437>fast <00:17:01.610>results. <00:17:01.782>They <00:17:01.955>just

00:17:02.070 --> 00:17:02.080 align:start position:0%
<00:17:02.070>They <00:17:02.079>just

00:17:02.080 --> 00:17:06.429 align:start position:0%
<00:17:02.080>They <00:17:02.255>just <00:17:02.430>question <00:17:02.605>of <00:17:02.780>whether <00:17:02.955>they <00:17:03.130>can <00:17:03.305>trust <00:17:03.480>it.

00:17:06.439 --> 00:17:08.150 align:start position:0%
<00:17:06.439>So <00:17:07.839>um

00:17:08.150 --> 00:17:08.160 align:start position:0%
<00:17:08.150>So <00:17:08.159>um

00:17:08.160 --> 00:17:09.510 align:start position:0%
<00:17:08.160>So <00:17:08.767>um <00:17:09.375>today

00:17:09.510 --> 00:17:09.520 align:start position:0%
today

00:17:09.520 --> 00:17:12.150 align:start position:0%
<00:17:09.520>today <00:17:09.800>we're <00:17:10.079>with <00:17:10.359>over <00:17:10.639>1,000 <00:17:10.920>production

00:17:12.150 --> 00:17:12.160 align:start position:0%
<00:17:12.150>we're <00:17:12.152>with <00:17:12.154>over <00:17:12.156>1,000 <00:17:12.159>production

00:17:12.160 --> 00:17:14.710 align:start position:0%
<00:17:12.160>we're <00:17:12.315>with <00:17:12.471>over <00:17:12.626>1,000 <00:17:12.782>production <00:17:12.937>environments <00:17:13.093>that <00:17:13.248>are <00:17:13.404>deployed <00:17:13.560>monitored

00:17:14.710 --> 00:17:14.720 align:start position:0%
<00:17:14.710>environments <00:17:14.712>that <00:17:14.714>are <00:17:14.716>deployed <00:17:14.719>monitored

00:17:14.720 --> 00:17:17.630 align:start position:0%
<00:17:14.720>environments <00:17:14.875>that <00:17:15.031>are <00:17:15.186>deployed <00:17:15.342>monitored <00:17:15.497>constantly <00:17:15.653>being <00:17:15.808>investigated <00:17:15.964>with <00:17:16.120>Ollie

00:17:17.630 --> 00:17:17.640 align:start position:0%
<00:17:17.630>constantly <00:17:17.632>being <00:17:17.634>investigated <00:17:17.636>with <00:17:17.639>Ollie

00:17:17.640 --> 00:17:19.829 align:start position:0%
<00:17:17.640>constantly <00:17:17.780>being <00:17:17.920>investigated <00:17:18.060>with <00:17:18.200>Ollie <00:17:18.340>whether <00:17:18.480>it's <00:17:18.620>scheduled <00:17:18.760>tasks <00:17:18.900>that <00:17:19.040>are

00:17:19.829 --> 00:17:19.839 align:start position:0%
<00:17:19.829>whether <00:17:19.830>it's <00:17:19.832>scheduled <00:17:19.834>tasks <00:17:19.836>that <00:17:19.838>are

00:17:19.839 --> 00:17:21.429 align:start position:0%
<00:17:19.839>whether <00:17:19.979>it's <00:17:20.118>scheduled <00:17:20.259>tasks <00:17:20.398>that <00:17:20.539>are <00:17:20.678>being <00:17:20.819>automated, <00:17:20.958>reports <00:17:21.098>that <00:17:21.239>are

00:17:21.429 --> 00:17:21.439 align:start position:0%
<00:17:21.429>being <00:17:21.431>automated, <00:17:21.433>reports <00:17:21.435>that <00:17:21.438>are

00:17:21.439 --> 00:17:23.189 align:start position:0%
<00:17:21.439>being <00:17:21.672>automated, <00:17:21.905>reports <00:17:22.139>that <00:17:22.372>are <00:17:22.605>created. <00:17:22.839>Um

00:17:23.189 --> 00:17:23.199 align:start position:0%
<00:17:23.189>created. <00:17:23.198>Um

00:17:23.199 --> 00:17:24.710 align:start position:0%
<00:17:23.199>created. <00:17:23.393>Um <00:17:23.587>So, <00:17:23.781>obviously <00:17:23.976>production <00:17:24.170>issues <00:17:24.364>that <00:17:24.558>are

00:17:24.710 --> 00:17:24.720 align:start position:0%
<00:17:24.710>So, <00:17:24.711>obviously <00:17:24.713>production <00:17:24.715>issues <00:17:24.717>that <00:17:24.719>are

00:17:24.720 --> 00:17:27.270 align:start position:0%
<00:17:24.720>So, <00:17:24.860>obviously <00:17:25.000>production <00:17:25.140>issues <00:17:25.280>that <00:17:25.420>are <00:17:25.560>being <00:17:25.700>investigated, <00:17:25.839>um <00:17:25.980>cloud <00:17:26.120>cost

00:17:27.270 --> 00:17:27.280 align:start position:0%
<00:17:27.270>being <00:17:27.272>investigated, <00:17:27.274>um <00:17:27.276>cloud <00:17:27.279>cost

00:17:27.280 --> 00:17:29.630 align:start position:0%
<00:17:27.280>being <00:17:27.407>investigated, <00:17:27.534>um <00:17:27.661>cloud <00:17:27.789>cost <00:17:27.916>optimization <00:17:28.043>use <00:17:28.170>cases <00:17:28.298>like <00:17:28.425>how <00:17:28.552>much <00:17:28.680>am

00:17:29.630 --> 00:17:29.640 align:start position:0%
<00:17:29.630>optimization <00:17:29.631>use <00:17:29.633>cases <00:17:29.634>like <00:17:29.636>how <00:17:29.637>much <00:17:29.639>am

00:17:29.640 --> 00:17:31.910 align:start position:0%
<00:17:29.640>optimization <00:17:29.740>use <00:17:29.840>cases <00:17:29.940>like <00:17:30.040>how <00:17:30.140>much <00:17:30.240>am <00:17:30.340>I <00:17:30.440>paying <00:17:30.540>for <00:17:30.640>these <00:17:30.740>resources? <00:17:30.840>How <00:17:30.940>how <00:17:31.040>do

00:17:31.910 --> 00:17:31.920 align:start position:0%
<00:17:31.910>I <00:17:31.911>paying <00:17:31.912>for <00:17:31.913>these <00:17:31.915>resources? <00:17:31.916>How <00:17:31.917>how <00:17:31.919>do

00:17:31.920 --> 00:17:34.070 align:start position:0%
<00:17:31.920>I <00:17:32.036>paying <00:17:32.153>for <00:17:32.270>these <00:17:32.386>resources? <00:17:32.503>How <00:17:32.620>how <00:17:32.736>do <00:17:32.853>these <00:17:32.970>resources <00:17:33.086>are <00:17:33.203>actually <00:17:33.320>affecting

00:17:34.070 --> 00:17:34.080 align:start position:0%
<00:17:34.070>these <00:17:34.072>resources <00:17:34.074>are <00:17:34.076>actually <00:17:34.079>affecting

00:17:34.080 --> 00:17:36.270 align:start position:0%
<00:17:34.080>these <00:17:34.280>resources <00:17:34.480>are <00:17:34.679>actually <00:17:34.879>affecting <00:17:35.080>my <00:17:35.280>production? <00:17:35.480>Um

00:17:36.270 --> 00:17:36.280 align:start position:0%
<00:17:36.270>my <00:17:36.274>production? <00:17:36.279>Um

00:17:36.280 --> 00:17:39.430 align:start position:0%
<00:17:36.280>my <00:17:36.480>production? <00:17:36.680>Um <00:17:36.879>product <00:17:37.080>analytics <00:17:37.280>Who <00:17:37.480>is <00:17:37.680>using

00:17:39.430 --> 00:17:39.440 align:start position:0%
<00:17:39.430>product <00:17:39.432>analytics <00:17:39.434>Who <00:17:39.436>is <00:17:39.439>using

00:17:39.440 --> 00:17:41.990 align:start position:0%
<00:17:39.440>product <00:17:39.547>analytics <00:17:39.655>Who <00:17:39.763>is <00:17:39.870>using <00:17:39.978>our <00:17:40.086>features? <00:17:40.193>How <00:17:40.301>do <00:17:40.409>they <00:17:40.516>use <00:17:40.624>it <00:17:40.732>and <00:17:40.840>so

00:17:41.990 --> 00:17:42.000 align:start position:0%
<00:17:41.990>our <00:17:41.991>features? <00:17:41.992>How <00:17:41.993>do <00:17:41.994>they <00:17:41.995>use <00:17:41.996>it <00:17:41.997>and <00:17:41.999>so

00:17:42.000 --> 00:17:44.830 align:start position:0%
<00:17:42.000>our <00:17:42.155>features? <00:17:42.311>How <00:17:42.466>do <00:17:42.622>they <00:17:42.777>use <00:17:42.933>it <00:17:43.088>and <00:17:43.244>so <00:17:43.400>on.

00:17:44.840 --> 00:17:46.750 align:start position:0%
<00:17:44.840>And <00:17:45.073>just <00:17:45.306>to <00:17:45.540>iterate <00:17:45.773>on <00:17:46.006>the <00:17:46.240>three

00:17:46.750 --> 00:17:46.760 align:start position:0%
<00:17:46.750>And <00:17:46.751>just <00:17:46.753>to <00:17:46.754>iterate <00:17:46.756>on <00:17:46.757>the <00:17:46.759>three

00:17:46.760 --> 00:17:49.070 align:start position:0%
<00:17:46.760>And <00:17:46.853>just <00:17:46.946>to <00:17:47.040>iterate <00:17:47.133>on <00:17:47.226>the <00:17:47.320>three <00:17:47.413>learnings <00:17:47.506>we <00:17:47.600>had <00:17:47.693>here. <00:17:47.786>Uh <00:17:47.879>and <00:17:47.973>and <00:17:48.066>by <00:17:48.160>the

00:17:49.070 --> 00:17:49.080 align:start position:0%
<00:17:49.070>learnings <00:17:49.071>we <00:17:49.072>had <00:17:49.073>here. <00:17:49.074>Uh <00:17:49.075>and <00:17:49.076>and <00:17:49.077>by <00:17:49.079>the

00:17:49.080 --> 00:17:50.630 align:start position:0%
<00:17:49.080>learnings <00:17:49.162>we <00:17:49.244>had <00:17:49.326>here. <00:17:49.408>Uh <00:17:49.490>and <00:17:49.572>and <00:17:49.654>by <00:17:49.736>the <00:17:49.818>way, <00:17:49.900>I <00:17:49.982>think <00:17:50.064>you <00:17:50.146>know, <00:17:50.228>it's <00:17:50.310>it's <00:17:50.392>new <00:17:50.475>for

00:17:50.630 --> 00:17:50.640 align:start position:0%
<00:17:50.630>way, <00:17:50.631>I <00:17:50.632>think <00:17:50.633>you <00:17:50.634>know, <00:17:50.635>it's <00:17:50.636>it's <00:17:50.637>new <00:17:50.639>for

00:17:50.640 --> 00:17:53.070 align:start position:0%
<00:17:50.640>way, <00:17:50.733>I <00:17:50.826>think <00:17:50.920>you <00:17:51.013>know, <00:17:51.106>it's <00:17:51.200>it's <00:17:51.293>new <00:17:51.386>for <00:17:51.480>everyone <00:17:51.573>agents, <00:17:51.666>right? <00:17:51.760>And <00:17:51.853>the <00:17:51.946>space <00:17:52.040>is

00:17:53.070 --> 00:17:53.080 align:start position:0%
<00:17:53.070>everyone <00:17:53.071>agents, <00:17:53.072>right? <00:17:53.074>And <00:17:53.076>the <00:17:53.077>space <00:17:53.079>is

00:17:53.080 --> 00:17:55.630 align:start position:0%
<00:17:53.080>everyone <00:17:53.196>agents, <00:17:53.313>right? <00:17:53.429>And <00:17:53.546>the <00:17:53.663>space <00:17:53.780>is <00:17:53.896>changing <00:17:54.013>rapidly. <00:17:54.129>Like <00:17:54.246>a <00:17:54.363>year <00:17:54.480>ago,

00:17:55.630 --> 00:17:55.640 align:start position:0%
<00:17:55.630>changing <00:17:55.631>rapidly. <00:17:55.633>Like <00:17:55.635>a <00:17:55.637>year <00:17:55.639>ago,

00:17:55.640 --> 00:17:57.790 align:start position:0%
<00:17:55.640>changing <00:17:55.780>rapidly. <00:17:55.920>Like <00:17:56.060>a <00:17:56.200>year <00:17:56.340>ago, <00:17:56.480>MCP <00:17:56.620>wasn't <00:17:56.760>a <00:17:56.900>thing <00:17:57.040>yet.

00:17:57.790 --> 00:17:57.800 align:start position:0%
<00:17:57.790>MCP <00:17:57.792>wasn't <00:17:57.794>a <00:17:57.796>thing <00:17:57.799>yet.

00:17:57.800 --> 00:18:00.190 align:start position:0%
<00:17:57.800>MCP <00:17:57.927>wasn't <00:17:58.054>a <00:17:58.181>thing <00:17:58.309>yet. <00:17:58.436>Today, <00:17:58.563>it's <00:17:58.690>already <00:17:58.818>old <00:17:58.945>the <00:17:59.072>old <00:17:59.200>news,

00:18:00.190 --> 00:18:00.200 align:start position:0%
<00:18:00.190>Today, <00:18:00.191>it's <00:18:00.193>already <00:18:00.194>old <00:18:00.196>the <00:18:00.197>old <00:18:00.199>news,

00:18:00.200 --> 00:18:01.990 align:start position:0%
<00:18:00.200>Today, <00:18:00.293>it's <00:18:00.386>already <00:18:00.480>old <00:18:00.573>the <00:18:00.666>old <00:18:00.760>news, <00:18:00.853>right? <00:18:00.946>Like <00:18:01.040>CLI <00:18:01.133>is <00:18:01.226>the <00:18:01.320>new <00:18:01.413>cool <00:18:01.506>kid <00:18:01.600>in

00:18:01.990 --> 00:18:02.000 align:start position:0%
<00:18:01.990>right? <00:18:01.991>Like <00:18:01.992>CLI <00:18:01.993>is <00:18:01.994>the <00:18:01.995>new <00:18:01.996>cool <00:18:01.997>kid <00:18:01.999>in

00:18:02.000 --> 00:18:04.190 align:start position:0%
<00:18:02.000>right? <00:18:02.155>Like <00:18:02.311>CLI <00:18:02.466>is <00:18:02.622>the <00:18:02.777>new <00:18:02.933>cool <00:18:03.088>kid <00:18:03.244>in <00:18:03.400>town.

00:18:04.190 --> 00:18:04.200 align:start position:0%
town.

00:18:04.200 --> 00:18:06.350 align:start position:0%
<00:18:04.200>town. <00:18:04.400>So, <00:18:04.600>you <00:18:04.800>constantly <00:18:05.000>we <00:18:05.200>constantly <00:18:05.400>I <00:18:05.600>think

00:18:06.350 --> 00:18:06.360 align:start position:0%
<00:18:06.350>So, <00:18:06.351>you <00:18:06.352>constantly <00:18:06.354>we <00:18:06.356>constantly <00:18:06.357>I <00:18:06.359>think

00:18:06.360 --> 00:18:07.950 align:start position:0%
<00:18:06.360>So, <00:18:06.500>you <00:18:06.639>constantly <00:18:06.780>we <00:18:06.919>constantly <00:18:07.060>I <00:18:07.199>think <00:18:07.340>learn <00:18:07.479>as <00:18:07.620>a <00:18:07.760>community.

00:18:07.950 --> 00:18:07.960 align:start position:0%
<00:18:07.950>learn <00:18:07.953>as <00:18:07.956>a <00:18:07.959>community.

00:18:07.960 --> 00:18:10.030 align:start position:0%
<00:18:07.960>learn <00:18:08.135>as <00:18:08.310>a <00:18:08.485>community. <00:18:08.660>Uh <00:18:08.835>and <00:18:09.010>I <00:18:09.185>think <00:18:09.360>sharing

00:18:10.030 --> 00:18:10.040 align:start position:0%
<00:18:10.030>Uh <00:18:10.032>and <00:18:10.034>I <00:18:10.036>think <00:18:10.039>sharing

00:18:10.040 --> 00:18:12.150 align:start position:0%
<00:18:10.040>Uh <00:18:10.180>and <00:18:10.320>I <00:18:10.460>think <00:18:10.600>sharing <00:18:10.740>what <00:18:10.879>works <00:18:11.020>and <00:18:11.159>most <00:18:11.300>importantly <00:18:11.440>what

00:18:12.150 --> 00:18:12.160 align:start position:0%
<00:18:12.150>what <00:18:12.151>works <00:18:12.153>and <00:18:12.155>most <00:18:12.157>importantly <00:18:12.159>what

00:18:12.160 --> 00:18:13.750 align:start position:0%
<00:18:12.160>what <00:18:12.267>works <00:18:12.375>and <00:18:12.483>most <00:18:12.590>importantly <00:18:12.698>what <00:18:12.806>didn't <00:18:12.913>work <00:18:13.021>and <00:18:13.129>what <00:18:13.236>did <00:18:13.344>you <00:18:13.452>learn, <00:18:13.560>I

00:18:13.750 --> 00:18:13.760 align:start position:0%
<00:18:13.750>didn't <00:18:13.751>work <00:18:13.752>and <00:18:13.753>what <00:18:13.755>did <00:18:13.756>you <00:18:13.757>learn, <00:18:13.759>I

00:18:13.760 --> 00:18:15.710 align:start position:0%
<00:18:13.760>didn't <00:18:13.860>work <00:18:13.960>and <00:18:14.060>what <00:18:14.160>did <00:18:14.260>you <00:18:14.360>learn, <00:18:14.460>I <00:18:14.560>think <00:18:14.660>it's <00:18:14.760>super <00:18:14.860>important. <00:18:14.960>Um <00:18:15.060>so, <00:18:15.160>just

00:18:15.710 --> 00:18:15.720 align:start position:0%
<00:18:15.710>think <00:18:15.711>it's <00:18:15.713>super <00:18:15.714>important. <00:18:15.716>Um <00:18:15.717>so, <00:18:15.719>just

00:18:15.720 --> 00:18:17.710 align:start position:0%
<00:18:15.720>think <00:18:15.836>it's <00:18:15.953>super <00:18:16.070>important. <00:18:16.186>Um <00:18:16.303>so, <00:18:16.420>just <00:18:16.536>to <00:18:16.653>summarize <00:18:16.770>three <00:18:16.886>of <00:18:17.003>our <00:18:17.120>learnings,

00:18:17.710 --> 00:18:17.720 align:start position:0%
<00:18:17.710>to <00:18:17.711>summarize <00:18:17.713>three <00:18:17.715>of <00:18:17.717>our <00:18:17.719>learnings,

00:18:17.720 --> 00:18:19.670 align:start position:0%
<00:18:17.720>to <00:18:17.847>summarize <00:18:17.974>three <00:18:18.101>of <00:18:18.229>our <00:18:18.356>learnings, <00:18:18.483>obviously <00:18:18.610>there <00:18:18.738>is <00:18:18.865>a <00:18:18.992>lot <00:18:19.120>more.

00:18:19.670 --> 00:18:19.680 align:start position:0%
<00:18:19.670>obviously <00:18:19.671>there <00:18:19.673>is <00:18:19.675>a <00:18:19.677>lot <00:18:19.679>more.

00:18:19.680 --> 00:18:21.070 align:start position:0%
<00:18:19.680>obviously <00:18:19.888>there <00:18:20.097>is <00:18:20.305>a <00:18:20.514>lot <00:18:20.722>more. <00:18:20.931>Um

00:18:21.070 --> 00:18:21.080 align:start position:0%
Um

00:18:21.080 --> 00:18:22.990 align:start position:0%
<00:18:21.080>Um <00:18:22.480>One,

00:18:22.990 --> 00:18:23.000 align:start position:0%
One,

00:18:23.000 --> 00:18:24.590 align:start position:0%
<00:18:23.000>One, <00:18:23.200>changing <00:18:23.400>a <00:18:23.600>model <00:18:23.800>when <00:18:24.000>there <00:18:24.200>there's <00:18:24.400>a

00:18:24.590 --> 00:18:24.600 align:start position:0%
<00:18:24.590>changing <00:18:24.591>a <00:18:24.592>model <00:18:24.594>when <00:18:24.596>there <00:18:24.597>there's <00:18:24.599>a

00:18:24.600 --> 00:18:26.270 align:start position:0%
<00:18:24.600>changing <00:18:24.699>a <00:18:24.800>model <00:18:24.899>when <00:18:25.000>there <00:18:25.100>there's <00:18:25.199>a <00:18:25.300>new <00:18:25.399>model <00:18:25.500>that <00:18:25.600>gets <00:18:25.699>released <00:18:25.800>or <00:18:25.899>when <00:18:26.000>you

00:18:26.270 --> 00:18:26.280 align:start position:0%
<00:18:26.270>new <00:18:26.271>model <00:18:26.272>that <00:18:26.273>gets <00:18:26.275>released <00:18:26.276>or <00:18:26.277>when <00:18:26.279>you

00:18:26.280 --> 00:18:27.710 align:start position:0%
<00:18:26.280>new <00:18:26.371>model <00:18:26.463>that <00:18:26.555>gets <00:18:26.647>released <00:18:26.739>or <00:18:26.831>when <00:18:26.923>you <00:18:27.015>want <00:18:27.107>to <00:18:27.199>upgrade <00:18:27.291>the <00:18:27.383>version <00:18:27.475>of <00:18:27.567>an

00:18:27.710 --> 00:18:27.720 align:start position:0%
<00:18:27.710>want <00:18:27.711>to <00:18:27.713>upgrade <00:18:27.714>the <00:18:27.716>version <00:18:27.717>of <00:18:27.719>an

00:18:27.720 --> 00:18:29.110 align:start position:0%
<00:18:27.720>want <00:18:27.876>to <00:18:28.032>upgrade <00:18:28.189>the <00:18:28.345>version <00:18:28.501>of <00:18:28.658>an <00:18:28.814>existing <00:18:28.971>model,

00:18:29.110 --> 00:18:29.120 align:start position:0%
<00:18:29.110>existing <00:18:29.119>model,

00:18:29.120 --> 00:18:30.950 align:start position:0%
<00:18:29.120>existing <00:18:29.294>model, <00:18:29.469>don't <00:18:29.645>treat <00:18:29.820>it <00:18:29.995>as <00:18:30.169>a <00:18:30.344>simple <00:18:30.520>string

00:18:30.950 --> 00:18:30.960 align:start position:0%
<00:18:30.950>don't <00:18:30.951>treat <00:18:30.953>it <00:18:30.954>as <00:18:30.956>a <00:18:30.957>simple <00:18:30.959>string

00:18:30.960 --> 00:18:33.870 align:start position:0%
<00:18:30.960>don't <00:18:31.067>treat <00:18:31.175>it <00:18:31.283>as <00:18:31.390>a <00:18:31.498>simple <00:18:31.606>string <00:18:31.713>replacement. <00:18:31.821>Treat <00:18:31.929>it <00:18:32.036>seriously <00:18:32.144>as <00:18:32.252>a <00:18:32.360>new

00:18:33.870 --> 00:18:33.880 align:start position:0%
<00:18:33.870>replacement. <00:18:33.871>Treat <00:18:33.873>it <00:18:33.874>seriously <00:18:33.876>as <00:18:33.877>a <00:18:33.879>new

00:18:33.880 --> 00:18:36.030 align:start position:0%
<00:18:33.880>replacement. <00:18:33.996>Treat <00:18:34.113>it <00:18:34.230>seriously <00:18:34.346>as <00:18:34.463>a <00:18:34.580>new <00:18:34.696>feature. <00:18:34.813>You <00:18:34.930>need <00:18:35.046>to <00:18:35.163>thoroughly <00:18:35.280>test,

00:18:36.030 --> 00:18:36.040 align:start position:0%
<00:18:36.030>feature. <00:18:36.031>You <00:18:36.033>need <00:18:36.035>to <00:18:36.037>thoroughly <00:18:36.039>test,

00:18:36.040 --> 00:18:37.510 align:start position:0%
<00:18:36.040>feature. <00:18:36.260>You <00:18:36.481>need <00:18:36.701>to <00:18:36.922>thoroughly <00:18:37.142>test, <00:18:37.363>evaluate,

00:18:37.510 --> 00:18:37.520 align:start position:0%
evaluate,

00:18:37.520 --> 00:18:40.030 align:start position:0%
<00:18:37.520>evaluate, <00:18:37.870>deploy <00:18:38.220>safely, <00:18:38.570>and <00:18:38.920>monitor.

00:18:40.030 --> 00:18:40.040 align:start position:0%
<00:18:40.030>deploy <00:18:40.033>safely, <00:18:40.036>and <00:18:40.039>monitor.

00:18:40.040 --> 00:18:41.870 align:start position:0%
<00:18:40.040>deploy <00:18:40.273>safely, <00:18:40.506>and <00:18:40.740>monitor. <00:18:40.973>Um <00:18:41.206>number <00:18:41.440>two,

00:18:41.870 --> 00:18:41.880 align:start position:0%
<00:18:41.870>Um <00:18:41.874>number <00:18:41.879>two,

00:18:41.880 --> 00:18:43.910 align:start position:0%
<00:18:41.880>Um <00:18:42.035>number <00:18:42.191>two, <00:18:42.346>it's <00:18:42.502>really <00:18:42.657>fun <00:18:42.813>to <00:18:42.968>track <00:18:43.124>activity <00:18:43.280>like

00:18:43.910 --> 00:18:43.920 align:start position:0%
<00:18:43.910>it's <00:18:43.911>really <00:18:43.913>fun <00:18:43.914>to <00:18:43.916>track <00:18:43.917>activity <00:18:43.919>like

00:18:43.920 --> 00:18:46.150 align:start position:0%
<00:18:43.920>it's <00:18:44.020>really <00:18:44.120>fun <00:18:44.220>to <00:18:44.320>track <00:18:44.420>activity <00:18:44.520>like <00:18:44.620>how <00:18:44.720>many <00:18:44.820>user <00:18:44.920>sessions <00:18:45.020>we <00:18:45.120>have, <00:18:45.220>how <00:18:45.320>many

00:18:46.150 --> 00:18:46.160 align:start position:0%
<00:18:46.150>how <00:18:46.151>many <00:18:46.152>user <00:18:46.153>sessions <00:18:46.155>we <00:18:46.156>have, <00:18:46.157>how <00:18:46.159>many

00:18:46.160 --> 00:18:47.990 align:start position:0%
<00:18:46.160>how <00:18:46.267>many <00:18:46.375>user <00:18:46.483>sessions <00:18:46.590>we <00:18:46.698>have, <00:18:46.806>how <00:18:46.913>many <00:18:47.021>tokens <00:18:47.129>were <00:18:47.236>consumed. <00:18:47.344>It's <00:18:47.452>great, <00:18:47.560>but

00:18:47.990 --> 00:18:48.000 align:start position:0%
<00:18:47.990>tokens <00:18:47.991>were <00:18:47.993>consumed. <00:18:47.995>It's <00:18:47.997>great, <00:18:47.999>but

00:18:48.000 --> 00:18:49.550 align:start position:0%
<00:18:48.000>tokens <00:18:48.174>were <00:18:48.348>consumed. <00:18:48.523>It's <00:18:48.697>great, <00:18:48.871>but <00:18:49.046>it's <00:18:49.220>not <00:18:49.395>enough.

00:18:49.550 --> 00:18:49.560 align:start position:0%
<00:18:49.550>it's <00:18:49.554>not <00:18:49.559>enough.

00:18:49.560 --> 00:18:52.430 align:start position:0%
<00:18:49.560>it's <00:18:49.793>not <00:18:50.026>enough. <00:18:50.260>Monitoring <00:18:50.493>cache <00:18:50.726>hit <00:18:50.960>ratio

00:18:52.430 --> 00:18:52.440 align:start position:0%
<00:18:52.430>Monitoring <00:18:52.433>cache <00:18:52.436>hit <00:18:52.439>ratio

00:18:52.440 --> 00:18:54.030 align:start position:0%
<00:18:52.440>Monitoring <00:18:52.595>cache <00:18:52.751>hit <00:18:52.906>ratio <00:18:53.062>essentially <00:18:53.217>allows <00:18:53.373>you <00:18:53.528>to <00:18:53.684>track <00:18:53.840>the

00:18:54.030 --> 00:18:54.040 align:start position:0%
<00:18:54.030>essentially <00:18:54.031>allows <00:18:54.033>you <00:18:54.035>to <00:18:54.037>track <00:18:54.039>the

00:18:54.040 --> 00:18:56.430 align:start position:0%
<00:18:54.040>essentially <00:18:54.167>allows <00:18:54.294>you <00:18:54.421>to <00:18:54.549>track <00:18:54.676>the <00:18:54.803>effect <00:18:54.930>the <00:18:55.058>effectiveness <00:18:55.185>or <00:18:55.312>the <00:18:55.440>cost

00:18:56.430 --> 00:18:56.440 align:start position:0%
<00:18:56.430>effect <00:18:56.431>the <00:18:56.433>effectiveness <00:18:56.435>or <00:18:56.437>the <00:18:56.439>cost

00:18:56.440 --> 00:19:00.470 align:start position:0%
<00:18:56.440>effect <00:18:56.556>the <00:18:56.673>effectiveness <00:18:56.790>or <00:18:56.906>the <00:18:57.023>cost <00:18:57.140>efficiency <00:18:57.256>of <00:18:57.373>your <00:18:57.490>agent. <00:18:57.606>Uh <00:18:57.723>and <00:18:57.840>lastly,

00:19:00.470 --> 00:19:00.480 align:start position:0%
<00:19:00.470>efficiency <00:19:00.471>of <00:19:00.473>your <00:19:00.474>agent. <00:19:00.476>Uh <00:19:00.477>and <00:19:00.479>lastly,

00:19:00.480 --> 00:19:01.950 align:start position:0%
<00:19:00.480>efficiency <00:19:00.645>of <00:19:00.810>your <00:19:00.976>agent. <00:19:01.141>Uh <00:19:01.306>and <00:19:01.472>lastly, <00:19:01.637>UX <00:19:01.803>matters.

00:19:01.950 --> 00:19:01.960 align:start position:0%
<00:19:01.950>UX <00:19:01.959>matters.

00:19:01.960 --> 00:19:03.830 align:start position:0%
<00:19:01.960>UX <00:19:02.160>matters. <00:19:02.360>Not <00:19:02.560>only <00:19:02.760>in <00:19:02.960>the <00:19:03.160>traditional <00:19:03.360>web

00:19:03.830 --> 00:19:03.840 align:start position:0%
<00:19:03.830>Not <00:19:03.831>only <00:19:03.833>in <00:19:03.835>the <00:19:03.837>traditional <00:19:03.839>web

00:19:03.840 --> 00:19:06.430 align:start position:0%
<00:19:03.840>Not <00:19:03.980>only <00:19:04.120>in <00:19:04.260>the <00:19:04.399>traditional <00:19:04.540>web <00:19:04.679>application <00:19:04.820>world, <00:19:04.959>but <00:19:05.100>moreover <00:19:05.240>actually

00:19:06.430 --> 00:19:06.440 align:start position:0%
<00:19:06.430>application <00:19:06.432>world, <00:19:06.434>but <00:19:06.436>moreover <00:19:06.439>actually

00:19:06.440 --> 00:19:08.830 align:start position:0%
<00:19:06.440>application <00:19:06.547>world, <00:19:06.655>but <00:19:06.763>moreover <00:19:06.870>actually <00:19:06.978>in <00:19:07.086>the <00:19:07.193>in <00:19:07.301>the <00:19:07.409>new <00:19:07.516>agentic <00:19:07.624>world. <00:19:07.732>And <00:19:07.840>And

00:19:08.830 --> 00:19:08.840 align:start position:0%
<00:19:08.830>in <00:19:08.831>the <00:19:08.832>in <00:19:08.833>the <00:19:08.834>new <00:19:08.835>agentic <00:19:08.836>world. <00:19:08.837>And <00:19:08.839>And

00:19:08.840 --> 00:19:11.030 align:start position:0%
<00:19:08.840>in <00:19:08.927>the <00:19:09.014>in <00:19:09.102>the <00:19:09.189>new <00:19:09.277>agentic <00:19:09.365>world. <00:19:09.452>And <00:19:09.540>And <00:19:09.627>as <00:19:09.715>we <00:19:09.802>see <00:19:09.889>more <00:19:09.977>users <00:19:10.064>using <00:19:10.152>the <00:19:10.240>agentic

00:19:11.030 --> 00:19:11.040 align:start position:0%
<00:19:11.030>as <00:19:11.031>we <00:19:11.032>see <00:19:11.033>more <00:19:11.035>users <00:19:11.036>using <00:19:11.037>the <00:19:11.039>agentic

00:19:11.040 --> 00:19:12.150 align:start position:0%
<00:19:11.040>as <00:19:11.164>we <00:19:11.289>see <00:19:11.414>more <00:19:11.539>users <00:19:11.664>using <00:19:11.789>the <00:19:11.914>agentic <00:19:12.039>interface,

00:19:12.150 --> 00:19:12.160 align:start position:0%
interface,

00:19:12.160 --> 00:19:13.950 align:start position:0%
<00:19:12.160>interface, <00:19:12.393>and <00:19:12.626>I <00:19:12.860>think <00:19:13.093>that's <00:19:13.326>another <00:19:13.560>something

00:19:13.950 --> 00:19:13.960 align:start position:0%
<00:19:13.950>and <00:19:13.951>I <00:19:13.953>think <00:19:13.955>that's <00:19:13.957>another <00:19:13.959>something

00:19:13.960 --> 00:19:15.350 align:start position:0%
<00:19:13.960>and <00:19:14.064>I <00:19:14.168>think <00:19:14.272>that's <00:19:14.377>another <00:19:14.481>something <00:19:14.585>else <00:19:14.689>that's <00:19:14.793>interesting. <00:19:14.898>We <00:19:15.002>see <00:19:15.106>not <00:19:15.211>only

00:19:15.350 --> 00:19:15.360 align:start position:0%
<00:19:15.350>else <00:19:15.351>that's <00:19:15.352>interesting. <00:19:15.354>We <00:19:15.356>see <00:19:15.357>not <00:19:15.359>only

00:19:15.360 --> 00:19:18.070 align:start position:0%
<00:19:15.360>else <00:19:15.487>that's <00:19:15.614>interesting. <00:19:15.741>We <00:19:15.869>see <00:19:15.996>not <00:19:16.123>only <00:19:16.250>developers <00:19:16.378>using <00:19:16.505>cloud <00:19:16.632>these <00:19:16.760>days,

00:19:18.070 --> 00:19:18.080 align:start position:0%
<00:19:18.070>developers <00:19:18.072>using <00:19:18.074>cloud <00:19:18.076>these <00:19:18.079>days,

00:19:18.080 --> 00:19:19.590 align:start position:0%
<00:19:18.080>developers <00:19:18.203>using <00:19:18.327>cloud <00:19:18.450>these <00:19:18.574>days, <00:19:18.697>right? <00:19:18.821>Like <00:19:18.944>you <00:19:19.068>can <00:19:19.191>see <00:19:19.315>marketing <00:19:19.438>teams

00:19:19.590 --> 00:19:19.600 align:start position:0%
<00:19:19.590>right? <00:19:19.591>Like <00:19:19.592>you <00:19:19.594>can <00:19:19.596>see <00:19:19.597>marketing <00:19:19.599>teams

00:19:19.600 --> 00:19:21.110 align:start position:0%
<00:19:19.600>right? <00:19:19.704>Like <00:19:19.809>you <00:19:19.913>can <00:19:20.018>see <00:19:20.122>marketing <00:19:20.227>teams <00:19:20.331>and <00:19:20.436>finance <00:19:20.540>team. <00:19:20.645>Like <00:19:20.749>everyone <00:19:20.854>is <00:19:20.958>now

00:19:21.110 --> 00:19:21.120 align:start position:0%
<00:19:21.110>and <00:19:21.111>finance <00:19:21.112>team. <00:19:21.114>Like <00:19:21.116>everyone <00:19:21.117>is <00:19:21.119>now

00:19:21.120 --> 00:19:23.550 align:start position:0%
<00:19:21.120>and <00:19:21.260>finance <00:19:21.399>team. <00:19:21.540>Like <00:19:21.679>everyone <00:19:21.820>is <00:19:21.959>now <00:19:22.100>using <00:19:22.239>these <00:19:22.379>tools. <00:19:22.520>Um

00:19:23.550 --> 00:19:23.560 align:start position:0%
<00:19:23.550>using <00:19:23.553>these <00:19:23.556>tools. <00:19:23.559>Um

00:19:23.560 --> 00:19:25.950 align:start position:0%
<00:19:23.560>using <00:19:23.676>these <00:19:23.793>tools. <00:19:23.909>Um <00:19:24.026>So <00:19:24.143>UX <00:19:24.260>matters <00:19:24.376>a <00:19:24.493>lot <00:19:24.610>and <00:19:24.726>we <00:19:24.843>need <00:19:24.960>to

00:19:25.950 --> 00:19:25.960 align:start position:0%
<00:19:25.950>So <00:19:25.951>UX <00:19:25.952>matters <00:19:25.953>a <00:19:25.954>lot <00:19:25.955>and <00:19:25.956>we <00:19:25.957>need <00:19:25.959>to

00:19:25.960 --> 00:19:27.590 align:start position:0%
<00:19:25.960>So <00:19:26.115>UX <00:19:26.271>matters <00:19:26.426>a <00:19:26.582>lot <00:19:26.737>and <00:19:26.893>we <00:19:27.048>need <00:19:27.204>to <00:19:27.360>rethink

00:19:27.590 --> 00:19:27.600 align:start position:0%
rethink

00:19:27.600 --> 00:19:29.190 align:start position:0%
<00:19:27.600>rethink <00:19:27.800>these <00:19:28.000>patterns <00:19:28.199>and <00:19:28.399>things <00:19:28.600>that <00:19:28.800>we <00:19:29.000>used

00:19:29.190 --> 00:19:29.200 align:start position:0%
<00:19:29.190>these <00:19:29.191>patterns <00:19:29.193>and <00:19:29.194>things <00:19:29.196>that <00:19:29.197>we <00:19:29.199>used

00:19:29.200 --> 00:19:30.430 align:start position:0%
<00:19:29.200>these <00:19:29.323>patterns <00:19:29.446>and <00:19:29.569>things <00:19:29.692>that <00:19:29.815>we <00:19:29.938>used <00:19:30.061>to <00:19:30.184>be <00:19:30.307>um

00:19:30.430 --> 00:19:30.440 align:start position:0%
<00:19:30.430>to <00:19:30.434>be <00:19:30.439>um

00:19:30.440 --> 00:19:32.430 align:start position:0%
<00:19:30.440>to <00:19:30.595>be <00:19:30.751>um <00:19:30.906>we <00:19:31.062>used <00:19:31.217>to <00:19:31.373>do <00:19:31.528>in <00:19:31.684>the <00:19:31.840>past.

00:19:32.430 --> 00:19:32.440 align:start position:0%
<00:19:32.430>we <00:19:32.431>used <00:19:32.433>to <00:19:32.434>do <00:19:32.436>in <00:19:32.437>the <00:19:32.439>past.

00:19:32.440 --> 00:19:35.270 align:start position:0%
<00:19:32.440>we <00:19:32.547>used <00:19:32.655>to <00:19:32.763>do <00:19:32.870>in <00:19:32.978>the <00:19:33.086>past. <00:19:33.193>Um <00:19:33.301>so <00:19:33.409>hopefully <00:19:33.516>it <00:19:33.624>was <00:19:33.732>interesting. <00:19:33.840>Um

00:19:35.270 --> 00:19:35.280 align:start position:0%
<00:19:35.270>Um <00:19:35.271>so <00:19:35.273>hopefully <00:19:35.274>it <00:19:35.276>was <00:19:35.277>interesting. <00:19:35.279>Um

00:19:35.280 --> 00:19:37.550 align:start position:0%
<00:19:35.280>Um <00:19:35.373>so <00:19:35.466>hopefully <00:19:35.560>it <00:19:35.653>was <00:19:35.746>interesting. <00:19:35.840>Um <00:19:35.933>I'll <00:19:36.026>be <00:19:36.120>more <00:19:36.213>than <00:19:36.306>happy <00:19:36.399>to <00:19:36.493>chat <00:19:36.586>over <00:19:36.680>our

00:19:37.550 --> 00:19:37.560 align:start position:0%
<00:19:37.550>I'll <00:19:37.551>be <00:19:37.552>more <00:19:37.553>than <00:19:37.554>happy <00:19:37.555>to <00:19:37.556>chat <00:19:37.557>over <00:19:37.559>our

00:19:37.560 --> 00:19:39.190 align:start position:0%
<00:19:37.560>I'll <00:19:37.659>be <00:19:37.760>more <00:19:37.860>than <00:19:37.960>happy <00:19:38.060>to <00:19:38.159>chat <00:19:38.260>over <00:19:38.360>our <00:19:38.460>booth <00:19:38.560>if <00:19:38.659>you <00:19:38.760>have <00:19:38.860>any <00:19:38.960>questions,

00:19:39.190 --> 00:19:39.200 align:start position:0%
<00:19:39.190>booth <00:19:39.191>if <00:19:39.193>you <00:19:39.195>have <00:19:39.197>any <00:19:39.199>questions,

00:19:39.200 --> 00:19:40.430 align:start position:0%
<00:19:39.200>booth <00:19:39.300>if <00:19:39.401>you <00:19:39.501>have <00:19:39.602>any <00:19:39.703>questions, <00:19:39.803>learnings, <00:19:39.904>things <00:19:40.005>you <00:19:40.105>would <00:19:40.206>like <00:19:40.307>to

00:19:40.430 --> 00:19:40.440 align:start position:0%
<00:19:40.430>learnings, <00:19:40.431>things <00:19:40.433>you <00:19:40.435>would <00:19:40.437>like <00:19:40.439>to

00:19:40.440 --> 00:19:41.310 align:start position:0%
<00:19:40.440>learnings, <00:19:40.570>things <00:19:40.701>you <00:19:40.831>would <00:19:40.962>like <00:19:41.092>to <00:19:41.223>share.

00:19:41.310 --> 00:19:41.320 align:start position:0%
share.

00:19:41.320 --> 00:19:43.588 align:start position:0%
<00:19:41.320>share. <00:19:41.600>Uh <00:19:41.879>thank <00:19:42.159>you <00:19:42.439>very <00:19:42.720>much.

00:19:43.588 --> 00:19:43.598 align:start position:0%
<00:19:43.588>Uh <00:19:43.590>thank <00:19:43.592>you <00:19:43.594>very <00:19:43.597>much.

00:19:43.598 --> 00:19:48.173 align:start position:0%
<00:19:43.598>Uh <00:19:43.831>thank <00:19:44.064>you <00:19:44.298>very <00:19:44.531>much. <00:19:44.764>&gt;&gt; <00:19:44.998>[applause]

00:19:48.183 --> 00:19:49.390 align:start position:0%
[applause]

00:19:49.390 --> 00:19:49.400 align:start position:0%
[applause]

00:19:49.400 --> 00:19:50.550 align:start position:0%
<00:19:49.400>[applause] <00:19:49.515>&gt;&gt; <00:19:49.630>I <00:19:49.745>think <00:19:49.860>we <00:19:49.975>have <00:19:50.090>a <00:19:50.205>few <00:19:50.320>minutes <00:19:50.435>for

00:19:50.550 --> 00:19:50.560 align:start position:0%
<00:19:50.550>&gt;&gt; <00:19:50.551>I <00:19:50.552>think <00:19:50.553>we <00:19:50.554>have <00:19:50.555>a <00:19:50.556>few <00:19:50.557>minutes <00:19:50.559>for

00:19:50.560 --> 00:19:52.710 align:start position:0%
<00:19:50.560>&gt;&gt; <00:19:50.715>I <00:19:50.871>think <00:19:51.026>we <00:19:51.182>have <00:19:51.337>a <00:19:51.493>few <00:19:51.648>minutes <00:19:51.804>for <00:19:51.960>questions.

00:19:52.710 --> 00:19:52.720 align:start position:0%
questions.

00:19:52.720 --> 00:19:58.230 align:start position:0%
<00:19:52.720>questions. <00:19:54.120>Okay.

00:19:58.240 --> 00:20:00.230 align:start position:0%
<00:19:58.240>&gt;&gt; <00:19:58.473>Thank <00:19:58.706>you <00:19:58.940>for <00:19:59.173>this <00:19:59.406>amazing <00:19:59.640>presentation.

00:20:00.230 --> 00:20:00.240 align:start position:0%
<00:20:00.230>&gt;&gt; <00:20:00.231>Thank <00:20:00.233>you <00:20:00.234>for <00:20:00.236>this <00:20:00.237>amazing <00:20:00.239>presentation.

00:20:00.240 --> 00:20:02.910 align:start position:0%
<00:20:00.240>&gt;&gt; <00:20:00.333>Thank <00:20:00.426>you <00:20:00.520>for <00:20:00.613>this <00:20:00.706>amazing <00:20:00.800>presentation. <00:20:00.893>Uh <00:20:00.986>we <00:20:01.080>have <00:20:01.173>a <00:20:01.266>question <00:20:01.360>for <00:20:01.453>you. <00:20:01.546>Once <00:20:01.640>you

00:20:02.910 --> 00:20:02.920 align:start position:0%
<00:20:02.910>Uh <00:20:02.911>we <00:20:02.912>have <00:20:02.913>a <00:20:02.914>question <00:20:02.915>for <00:20:02.916>you. <00:20:02.917>Once <00:20:02.919>you

00:20:02.920 --> 00:20:05.230 align:start position:0%
<00:20:02.920>Uh <00:20:03.007>we <00:20:03.095>have <00:20:03.182>a <00:20:03.270>question <00:20:03.357>for <00:20:03.445>you. <00:20:03.532>Once <00:20:03.620>you <00:20:03.707>change <00:20:03.795>a <00:20:03.882>model, <00:20:03.970>how <00:20:04.057>do <00:20:04.145>you <00:20:04.232>know <00:20:04.320>what

00:20:05.230 --> 00:20:05.240 align:start position:0%
<00:20:05.230>change <00:20:05.231>a <00:20:05.232>model, <00:20:05.233>how <00:20:05.235>do <00:20:05.236>you <00:20:05.237>know <00:20:05.239>what

00:20:05.240 --> 00:20:07.310 align:start position:0%
<00:20:05.240>change <00:20:05.367>a <00:20:05.494>model, <00:20:05.621>how <00:20:05.749>do <00:20:05.876>you <00:20:06.003>know <00:20:06.130>what <00:20:06.258>needs <00:20:06.385>to <00:20:06.512>be <00:20:06.640>updated?

00:20:07.310 --> 00:20:07.320 align:start position:0%
<00:20:07.310>needs <00:20:07.312>to <00:20:07.316>be <00:20:07.319>updated?

00:20:07.320 --> 00:20:09.550 align:start position:0%
<00:20:07.320>needs <00:20:07.447>to <00:20:07.574>be <00:20:07.701>updated? <00:20:07.829>&gt;&gt; <00:20:07.956>It's <00:20:08.083>a <00:20:08.210>really <00:20:08.338>really <00:20:08.465>good <00:20:08.592>question. <00:20:08.720>Um

00:20:09.550 --> 00:20:09.560 align:start position:0%
<00:20:09.550>&gt;&gt; <00:20:09.551>It's <00:20:09.552>a <00:20:09.553>really <00:20:09.555>really <00:20:09.556>good <00:20:09.557>question. <00:20:09.559>Um

00:20:09.560 --> 00:20:10.950 align:start position:0%
<00:20:09.560>&gt;&gt; <00:20:09.656>It's <00:20:09.752>a <00:20:09.848>really <00:20:09.944>really <00:20:10.041>good <00:20:10.137>question. <00:20:10.233>Um <00:20:10.329>So <00:20:10.426>first <00:20:10.522>and <00:20:10.618>foremost, <00:20:10.714>the <00:20:10.811>most

00:20:10.950 --> 00:20:10.960 align:start position:0%
<00:20:10.950>So <00:20:10.951>first <00:20:10.953>and <00:20:10.955>foremost, <00:20:10.957>the <00:20:10.959>most

00:20:10.960 --> 00:20:12.670 align:start position:0%
<00:20:10.960>So <00:20:11.087>first <00:20:11.214>and <00:20:11.341>foremost, <00:20:11.469>the <00:20:11.596>most <00:20:11.723>important <00:20:11.850>thing <00:20:11.978>is <00:20:12.105>to <00:20:12.232>have <00:20:12.360>evaluation

00:20:12.670 --> 00:20:12.680 align:start position:0%
<00:20:12.670>important <00:20:12.671>thing <00:20:12.673>is <00:20:12.675>to <00:20:12.677>have <00:20:12.679>evaluation

00:20:12.680 --> 00:20:15.190 align:start position:0%
<00:20:12.680>important <00:20:12.796>thing <00:20:12.913>is <00:20:13.030>to <00:20:13.146>have <00:20:13.263>evaluation <00:20:13.380>system <00:20:13.496>in <00:20:13.613>place. <00:20:13.730>What <00:20:13.846>that <00:20:13.963>What <00:20:14.080>does

00:20:15.190 --> 00:20:15.200 align:start position:0%
<00:20:15.190>system <00:20:15.191>in <00:20:15.193>place. <00:20:15.194>What <00:20:15.196>that <00:20:15.197>What <00:20:15.199>does

00:20:15.200 --> 00:20:17.830 align:start position:0%
<00:20:15.200>system <00:20:15.300>in <00:20:15.400>place. <00:20:15.500>What <00:20:15.600>that <00:20:15.700>What <00:20:15.800>does <00:20:15.900>that <00:20:16.000>mean? <00:20:16.100>That <00:20:16.200>means <00:20:16.300>we <00:20:16.400>have <00:20:16.500>a <00:20:16.600>system

00:20:17.830 --> 00:20:17.840 align:start position:0%
<00:20:17.830>that <00:20:17.831>mean? <00:20:17.832>That <00:20:17.833>means <00:20:17.835>we <00:20:17.836>have <00:20:17.837>a <00:20:17.839>system

00:20:17.840 --> 00:20:18.830 align:start position:0%
<00:20:17.840>that <00:20:17.951>mean? <00:20:18.062>That <00:20:18.174>means <00:20:18.285>we <00:20:18.396>have <00:20:18.508>a <00:20:18.619>system <00:20:18.731>with

00:20:18.830 --> 00:20:18.840 align:start position:0%
with

00:20:18.840 --> 00:20:21.390 align:start position:0%
<00:20:18.840>with <00:20:19.040>pre-made <00:20:19.240>test <00:20:19.439>cases <00:20:19.639>and <00:20:19.840>and <00:20:20.040>kind <00:20:20.240>of

00:20:21.390 --> 00:20:21.400 align:start position:0%
<00:20:21.390>pre-made <00:20:21.391>test <00:20:21.393>cases <00:20:21.394>and <00:20:21.396>and <00:20:21.397>kind <00:20:21.399>of

00:20:21.400 --> 00:20:23.790 align:start position:0%
<00:20:21.400>pre-made <00:20:21.540>test <00:20:21.680>cases <00:20:21.820>and <00:20:21.960>and <00:20:22.100>kind <00:20:22.240>of <00:20:22.380>sandbox <00:20:22.520>environments <00:20:22.660>of <00:20:22.800>production-like

00:20:23.790 --> 00:20:23.800 align:start position:0%
<00:20:23.790>sandbox <00:20:23.793>environments <00:20:23.796>of <00:20:23.799>production-like

00:20:23.800 --> 00:20:25.510 align:start position:0%
<00:20:23.800>sandbox <00:20:24.000>environments <00:20:24.200>of <00:20:24.399>production-like <00:20:24.600>use <00:20:24.800>real <00:20:25.000>use <00:20:25.200>cases.

00:20:25.510 --> 00:20:25.520 align:start position:0%
<00:20:25.510>use <00:20:25.513>real <00:20:25.516>use <00:20:25.519>cases.

00:20:25.520 --> 00:20:29.710 align:start position:0%
<00:20:25.520>use <00:20:25.647>real <00:20:25.774>use <00:20:25.901>cases. <00:20:26.029>Uh <00:20:26.156>and <00:20:26.283>for <00:20:26.410>each <00:20:26.538>eval <00:20:26.665>we <00:20:26.792>have <00:20:26.920>actually

00:20:29.710 --> 00:20:29.720 align:start position:0%
<00:20:29.710>Uh <00:20:29.711>and <00:20:29.712>for <00:20:29.713>each <00:20:29.715>eval <00:20:29.716>we <00:20:29.717>have <00:20:29.719>actually

00:20:29.720 --> 00:20:31.550 align:start position:0%
<00:20:29.720>Uh <00:20:29.813>and <00:20:29.906>for <00:20:30.000>each <00:20:30.093>eval <00:20:30.186>we <00:20:30.280>have <00:20:30.373>actually <00:20:30.466>I <00:20:30.560>let <00:20:30.653>let <00:20:30.746>me <00:20:30.840>actually <00:20:30.933>explain <00:20:31.026>because <00:20:31.120>I

00:20:31.550 --> 00:20:31.560 align:start position:0%
<00:20:31.550>I <00:20:31.551>let <00:20:31.552>let <00:20:31.553>me <00:20:31.555>actually <00:20:31.556>explain <00:20:31.557>because <00:20:31.559>I

00:20:31.560 --> 00:20:33.910 align:start position:0%
<00:20:31.560>I <00:20:31.653>let <00:20:31.746>let <00:20:31.840>me <00:20:31.933>actually <00:20:32.026>explain <00:20:32.120>because <00:20:32.213>I <00:20:32.306>think <00:20:32.399>eval <00:20:32.493>is <00:20:32.586>a <00:20:32.679>is <00:20:32.773>an <00:20:32.866>interesting <00:20:32.960>topic.

00:20:33.910 --> 00:20:33.920 align:start position:0%
<00:20:33.910>think <00:20:33.911>eval <00:20:33.912>is <00:20:33.913>a <00:20:33.915>is <00:20:33.916>an <00:20:33.917>interesting <00:20:33.919>topic.

00:20:33.920 --> 00:20:36.430 align:start position:0%
<00:20:33.920>think <00:20:34.013>eval <00:20:34.106>is <00:20:34.200>a <00:20:34.293>is <00:20:34.386>an <00:20:34.480>interesting <00:20:34.573>topic. <00:20:34.666>Um <00:20:34.760>evals <00:20:34.853>is <00:20:34.946>not <00:20:35.040>unit <00:20:35.133>testing. <00:20:35.226>And <00:20:35.320>this

00:20:36.430 --> 00:20:36.440 align:start position:0%
<00:20:36.430>Um <00:20:36.431>evals <00:20:36.432>is <00:20:36.433>not <00:20:36.435>unit <00:20:36.436>testing. <00:20:36.437>And <00:20:36.439>this

00:20:36.440 --> 00:20:38.230 align:start position:0%
<00:20:36.440>Um <00:20:36.540>evals <00:20:36.640>is <00:20:36.740>not <00:20:36.840>unit <00:20:36.940>testing. <00:20:37.040>And <00:20:37.140>this <00:20:37.240>is <00:20:37.340>important. <00:20:37.440>It <00:20:37.540>doesn't <00:20:37.640>like <00:20:37.740>a <00:20:37.840>test

00:20:38.230 --> 00:20:38.240 align:start position:0%
<00:20:38.230>is <00:20:38.231>important. <00:20:38.233>It <00:20:38.234>doesn't <00:20:38.236>like <00:20:38.237>a <00:20:38.239>test

00:20:38.240 --> 00:20:40.110 align:start position:0%
<00:20:38.240>is <00:20:38.395>important. <00:20:38.551>It <00:20:38.706>doesn't <00:20:38.862>like <00:20:39.017>a <00:20:39.173>test <00:20:39.328>pass <00:20:39.484>or <00:20:39.640>fails.

00:20:40.110 --> 00:20:40.120 align:start position:0%
<00:20:40.110>pass <00:20:40.114>or <00:20:40.119>fails.

00:20:40.120 --> 00:20:42.870 align:start position:0%
<00:20:40.120>pass <00:20:40.247>or <00:20:40.374>fails. <00:20:40.501>Right? <00:20:40.629>So <00:20:40.756>we <00:20:40.883>have <00:20:41.010>two <00:20:41.138>ways <00:20:41.265>to <00:20:41.392>score <00:20:41.520>our

00:20:42.870 --> 00:20:42.880 align:start position:0%
<00:20:42.870>Right? <00:20:42.871>So <00:20:42.872>we <00:20:42.873>have <00:20:42.874>two <00:20:42.875>ways <00:20:42.876>to <00:20:42.877>score <00:20:42.879>our

00:20:42.880 --> 00:20:45.590 align:start position:0%
<00:20:42.880>Right? <00:20:42.973>So <00:20:43.066>we <00:20:43.160>have <00:20:43.253>two <00:20:43.346>ways <00:20:43.440>to <00:20:43.533>score <00:20:43.626>our <00:20:43.720>agent <00:20:43.813>when <00:20:43.906>it <00:20:44.000>comes <00:20:44.093>to <00:20:44.186>eval. <00:20:44.280>One,

00:20:45.590 --> 00:20:45.600 align:start position:0%
<00:20:45.590>agent <00:20:45.591>when <00:20:45.592>it <00:20:45.594>comes <00:20:45.596>to <00:20:45.597>eval. <00:20:45.599>One,

00:20:45.600 --> 00:20:47.790 align:start position:0%
<00:20:45.600>agent <00:20:45.740>when <00:20:45.879>it <00:20:46.020>comes <00:20:46.159>to <00:20:46.300>eval. <00:20:46.439>One, <00:20:46.580>did <00:20:46.719>it <00:20:46.860>actually <00:20:47.000>achieve

00:20:47.790 --> 00:20:47.800 align:start position:0%
<00:20:47.790>did <00:20:47.793>it <00:20:47.796>actually <00:20:47.799>achieve

00:20:47.800 --> 00:20:49.430 align:start position:0%
<00:20:47.800>did <00:20:47.927>it <00:20:48.054>actually <00:20:48.181>achieve <00:20:48.309>the <00:20:48.436>the <00:20:48.563>end <00:20:48.690>goal? <00:20:48.818>So <00:20:48.945>for <00:20:49.072>example, <00:20:49.200>if

00:20:49.430 --> 00:20:49.440 align:start position:0%
<00:20:49.430>the <00:20:49.431>the <00:20:49.432>end <00:20:49.433>goal? <00:20:49.435>So <00:20:49.436>for <00:20:49.437>example, <00:20:49.439>if

00:20:49.440 --> 00:20:50.750 align:start position:0%
<00:20:49.440>the <00:20:49.530>the <00:20:49.621>end <00:20:49.712>goal? <00:20:49.802>So <00:20:49.893>for <00:20:49.984>example, <00:20:50.074>if <00:20:50.165>there's <00:20:50.256>production <00:20:50.346>issue <00:20:50.437>and <00:20:50.528>the <00:20:50.619>end

00:20:50.750 --> 00:20:50.760 align:start position:0%
<00:20:50.750>there's <00:20:50.751>production <00:20:50.753>issue <00:20:50.755>and <00:20:50.757>the <00:20:50.759>end

00:20:50.760 --> 00:20:52.630 align:start position:0%
<00:20:50.760>there's <00:20:50.860>production <00:20:50.960>issue <00:20:51.060>and <00:20:51.160>the <00:20:51.260>end <00:20:51.360>goal <00:20:51.460>is <00:20:51.560>to <00:20:51.660>find <00:20:51.760>the <00:20:51.860>root <00:20:51.960>cause, <00:20:52.060>this <00:20:52.160>is

00:20:52.630 --> 00:20:52.640 align:start position:0%
<00:20:52.630>goal <00:20:52.631>is <00:20:52.632>to <00:20:52.633>find <00:20:52.634>the <00:20:52.635>root <00:20:52.636>cause, <00:20:52.637>this <00:20:52.639>is

00:20:52.640 --> 00:20:54.510 align:start position:0%
<00:20:52.640>goal <00:20:52.767>is <00:20:52.894>to <00:20:53.021>find <00:20:53.149>the <00:20:53.276>root <00:20:53.403>cause, <00:20:53.530>this <00:20:53.658>is <00:20:53.785>the <00:20:53.912>success <00:20:54.040>criteria.

00:20:54.510 --> 00:20:54.520 align:start position:0%
<00:20:54.510>the <00:20:54.514>success <00:20:54.519>criteria.

00:20:54.520 --> 00:20:56.550 align:start position:0%
<00:20:54.520>the <00:20:54.660>success <00:20:54.800>criteria. <00:20:54.940>But <00:20:55.080>still, <00:20:55.220>think <00:20:55.360>about <00:20:55.500>it. <00:20:55.639>Even <00:20:55.780>if <00:20:55.920>the

00:20:56.550 --> 00:20:56.560 align:start position:0%
<00:20:56.550>But <00:20:56.551>still, <00:20:56.552>think <00:20:56.553>about <00:20:56.555>it. <00:20:56.556>Even <00:20:56.557>if <00:20:56.559>the

00:20:56.560 --> 00:20:59.110 align:start position:0%
<00:20:56.560>But <00:20:56.667>still, <00:20:56.775>think <00:20:56.883>about <00:20:56.990>it. <00:20:57.098>Even <00:20:57.206>if <00:20:57.313>the <00:20:57.421>agent <00:20:57.529>didn't <00:20:57.636>find <00:20:57.744>the <00:20:57.852>root <00:20:57.960>cause,

00:20:59.110 --> 00:20:59.120 align:start position:0%
<00:20:59.110>agent <00:20:59.111>didn't <00:20:59.113>find <00:20:59.115>the <00:20:59.117>root <00:20:59.119>cause,

00:20:59.120 --> 00:21:01.190 align:start position:0%
<00:20:59.120>agent <00:20:59.227>didn't <00:20:59.335>find <00:20:59.443>the <00:20:59.550>root <00:20:59.658>cause, <00:20:59.766>it <00:20:59.873>still <00:20:59.981>may <00:21:00.089>have <00:21:00.196>been <00:21:00.304>may <00:21:00.412>provided <00:21:00.520>some

00:21:01.190 --> 00:21:01.200 align:start position:0%
<00:21:01.190>it <00:21:01.191>still <00:21:01.192>may <00:21:01.193>have <00:21:01.195>been <00:21:01.196>may <00:21:01.197>provided <00:21:01.199>some

00:21:01.200 --> 00:21:02.950 align:start position:0%
<00:21:01.200>it <00:21:01.307>still <00:21:01.415>may <00:21:01.523>have <00:21:01.630>been <00:21:01.738>may <00:21:01.846>provided <00:21:01.953>some <00:21:02.061>valuable <00:21:02.169>insights, <00:21:02.276>right? <00:21:02.384>So <00:21:02.492>we <00:21:02.600>want

00:21:02.950 --> 00:21:02.960 align:start position:0%
<00:21:02.950>valuable <00:21:02.951>insights, <00:21:02.953>right? <00:21:02.955>So <00:21:02.957>we <00:21:02.959>want

00:21:02.960 --> 00:21:05.070 align:start position:0%
<00:21:02.960>valuable <00:21:03.087>insights, <00:21:03.214>right? <00:21:03.341>So <00:21:03.469>we <00:21:03.596>want <00:21:03.723>also <00:21:03.850>to <00:21:03.978>score <00:21:04.105>it <00:21:04.232>positively <00:21:04.360>or

00:21:05.070 --> 00:21:05.080 align:start position:0%
<00:21:05.070>also <00:21:05.071>to <00:21:05.073>score <00:21:05.075>it <00:21:05.077>positively <00:21:05.079>or

00:21:05.080 --> 00:21:06.910 align:start position:0%
<00:21:05.080>also <00:21:05.207>to <00:21:05.334>score <00:21:05.461>it <00:21:05.589>positively <00:21:05.716>or <00:21:05.843>negatively <00:21:05.970>based <00:21:06.098>on <00:21:06.225>some <00:21:06.352>insight <00:21:06.480>and

00:21:06.910 --> 00:21:06.920 align:start position:0%
<00:21:06.910>negatively <00:21:06.911>based <00:21:06.913>on <00:21:06.915>some <00:21:06.917>insight <00:21:06.919>and

00:21:06.920 --> 00:21:09.030 align:start position:0%
<00:21:06.920>negatively <00:21:07.075>based <00:21:07.231>on <00:21:07.386>some <00:21:07.542>insight <00:21:07.697>and <00:21:07.853>behaviors <00:21:08.008>throughout <00:21:08.164>the <00:21:08.320>process.

00:21:09.030 --> 00:21:09.040 align:start position:0%
<00:21:09.030>behaviors <00:21:09.033>throughout <00:21:09.036>the <00:21:09.039>process.

00:21:09.040 --> 00:21:11.790 align:start position:0%
<00:21:09.040>behaviors <00:21:09.147>throughout <00:21:09.255>the <00:21:09.363>process. <00:21:09.470>So, <00:21:09.578>that <00:21:09.686>uh <00:21:09.793>lands <00:21:09.901>us <00:21:10.009>at <00:21:10.116>a <00:21:10.224>place <00:21:10.332>where <00:21:10.440>by

00:21:11.790 --> 00:21:11.800 align:start position:0%
<00:21:11.790>So, <00:21:11.791>that <00:21:11.792>uh <00:21:11.793>lands <00:21:11.793>us <00:21:11.795>at <00:21:11.796>a <00:21:11.797>place <00:21:11.798>where <00:21:11.799>by

00:21:11.800 --> 00:21:13.310 align:start position:0%
<00:21:11.800>So, <00:21:11.875>that <00:21:11.951>uh <00:21:12.026>lands <00:21:12.101>us <00:21:12.177>at <00:21:12.253>a <00:21:12.328>place <00:21:12.404>where <00:21:12.479>by <00:21:12.554>the <00:21:12.630>end <00:21:12.706>of <00:21:12.781>running <00:21:12.857>all <00:21:12.932>of <00:21:13.008>our <00:21:13.083>evils, <00:21:13.158>we

00:21:13.310 --> 00:21:13.320 align:start position:0%
<00:21:13.310>the <00:21:13.311>end <00:21:13.312>of <00:21:13.313>running <00:21:13.314>all <00:21:13.315>of <00:21:13.316>our <00:21:13.317>evils, <00:21:13.319>we

00:21:13.320 --> 00:21:16.030 align:start position:0%
<00:21:13.320>the <00:21:13.402>end <00:21:13.484>of <00:21:13.567>running <00:21:13.649>all <00:21:13.731>of <00:21:13.814>our <00:21:13.896>evils, <00:21:13.978>we <00:21:14.061>have <00:21:14.143>a <00:21:14.225>total <00:21:14.308>pass <00:21:14.390>rate. <00:21:14.472>So, <00:21:14.555>how <00:21:14.637>many <00:21:14.720>of

00:21:16.030 --> 00:21:16.040 align:start position:0%
<00:21:16.030>have <00:21:16.031>a <00:21:16.032>total <00:21:16.033>pass <00:21:16.034>rate. <00:21:16.035>So, <00:21:16.036>how <00:21:16.037>many <00:21:16.039>of

00:21:16.040 --> 00:21:18.390 align:start position:0%
<00:21:16.040>have <00:21:16.127>a <00:21:16.215>total <00:21:16.302>pass <00:21:16.389>rate. <00:21:16.477>So, <00:21:16.565>how <00:21:16.652>many <00:21:16.740>of <00:21:16.827>the <00:21:16.915>use <00:21:17.002>cases <00:21:17.090>it <00:21:17.177>actually <00:21:17.264>got <00:21:17.352>the <00:21:17.440>the

00:21:18.390 --> 00:21:18.400 align:start position:0%
<00:21:18.390>the <00:21:18.391>use <00:21:18.392>cases <00:21:18.393>it <00:21:18.395>actually <00:21:18.396>got <00:21:18.397>the <00:21:18.399>the

00:21:18.400 --> 00:21:19.750 align:start position:0%
<00:21:18.400>the <00:21:18.535>use <00:21:18.670>cases <00:21:18.805>it <00:21:18.940>actually <00:21:19.075>got <00:21:19.210>the <00:21:19.345>the <00:21:19.480>holy <00:21:19.615>grail

00:21:19.750 --> 00:21:19.760 align:start position:0%
<00:21:19.750>holy <00:21:19.759>grail

00:21:19.760 --> 00:21:21.790 align:start position:0%
<00:21:19.760>holy <00:21:20.040>grail <00:21:20.320>and <00:21:20.600>actually <00:21:20.879>another <00:21:21.160>score.

00:21:21.790 --> 00:21:21.800 align:start position:0%
<00:21:21.790>and <00:21:21.793>actually <00:21:21.796>another <00:21:21.799>score.

00:21:21.800 --> 00:21:23.550 align:start position:0%
<00:21:21.800>and <00:21:21.940>actually <00:21:22.080>another <00:21:22.220>score. <00:21:22.360>With <00:21:22.500>these <00:21:22.639>two, <00:21:22.780>we <00:21:22.919>can <00:21:23.060>actually <00:21:23.200>score

00:21:23.550 --> 00:21:23.560 align:start position:0%
<00:21:23.550>With <00:21:23.551>these <00:21:23.553>two, <00:21:23.554>we <00:21:23.556>can <00:21:23.557>actually <00:21:23.559>score

00:21:23.560 --> 00:21:26.510 align:start position:0%
<00:21:23.560>With <00:21:23.653>these <00:21:23.746>two, <00:21:23.840>we <00:21:23.933>can <00:21:24.026>actually <00:21:24.120>score <00:21:24.213>and <00:21:24.306>tell <00:21:24.399>have <00:21:24.493>we <00:21:24.586>improved <00:21:24.679>or <00:21:24.773>not <00:21:24.866>or <00:21:24.960>have

00:21:26.510 --> 00:21:26.520 align:start position:0%
<00:21:26.510>and <00:21:26.511>tell <00:21:26.512>have <00:21:26.513>we <00:21:26.514>improved <00:21:26.515>or <00:21:26.516>not <00:21:26.517>or <00:21:26.519>have

00:21:26.520 --> 00:21:28.270 align:start position:0%
<00:21:26.520>and <00:21:26.636>tell <00:21:26.753>have <00:21:26.870>we <00:21:26.986>improved <00:21:27.103>or <00:21:27.220>not <00:21:27.336>or <00:21:27.453>have <00:21:27.570>we <00:21:27.686>actually <00:21:27.803>gone <00:21:27.920>backwards.

00:21:28.270 --> 00:21:28.280 align:start position:0%
<00:21:28.270>we <00:21:28.273>actually <00:21:28.276>gone <00:21:28.279>backwards.

00:21:28.280 --> 00:21:31.030 align:start position:0%
<00:21:28.280>we <00:21:28.407>actually <00:21:28.534>gone <00:21:28.661>backwards. <00:21:28.789>So, <00:21:28.916>once <00:21:29.043>we <00:21:29.170>have <00:21:29.298>these <00:21:29.425>results <00:21:29.552>in <00:21:29.680>place,

00:21:31.030 --> 00:21:31.040 align:start position:0%
<00:21:31.030>So, <00:21:31.031>once <00:21:31.032>we <00:21:31.033>have <00:21:31.035>these <00:21:31.036>results <00:21:31.037>in <00:21:31.039>place,

00:21:31.040 --> 00:21:33.150 align:start position:0%
<00:21:31.040>So, <00:21:31.133>once <00:21:31.226>we <00:21:31.320>have <00:21:31.413>these <00:21:31.506>results <00:21:31.600>in <00:21:31.693>place, <00:21:31.786>honestly, <00:21:31.879>it's <00:21:31.973>a <00:21:32.066>lot <00:21:32.159>of <00:21:32.253>trial <00:21:32.346>and <00:21:32.440>error

00:21:33.150 --> 00:21:33.160 align:start position:0%
<00:21:33.150>honestly, <00:21:33.151>it's <00:21:33.152>a <00:21:33.153>lot <00:21:33.155>of <00:21:33.156>trial <00:21:33.157>and <00:21:33.159>error

00:21:33.160 --> 00:21:35.910 align:start position:0%
<00:21:33.160>honestly, <00:21:33.267>it's <00:21:33.375>a <00:21:33.483>lot <00:21:33.590>of <00:21:33.698>trial <00:21:33.806>and <00:21:33.913>error <00:21:34.021>and <00:21:34.129>the <00:21:34.236>specific <00:21:34.344>developer <00:21:34.452>working <00:21:34.560>on

00:21:35.910 --> 00:21:35.920 align:start position:0%
<00:21:35.910>and <00:21:35.911>the <00:21:35.913>specific <00:21:35.915>developer <00:21:35.917>working <00:21:35.919>on

00:21:35.920 --> 00:21:38.790 align:start position:0%
<00:21:35.920>and <00:21:36.027>the <00:21:36.135>specific <00:21:36.243>developer <00:21:36.350>working <00:21:36.458>on <00:21:36.566>the <00:21:36.673>task <00:21:36.781>kind <00:21:36.889>of <00:21:36.996>get <00:21:37.104>the <00:21:37.212>feeling <00:21:37.320>of

00:21:38.790 --> 00:21:38.800 align:start position:0%
<00:21:38.790>the <00:21:38.791>task <00:21:38.792>kind <00:21:38.793>of <00:21:38.795>get <00:21:38.796>the <00:21:38.797>feeling <00:21:38.799>of

00:21:38.800 --> 00:21:40.870 align:start position:0%
<00:21:38.800>the <00:21:38.893>task <00:21:38.986>kind <00:21:39.080>of <00:21:39.173>get <00:21:39.266>the <00:21:39.360>feeling <00:21:39.453>of <00:21:39.546>how <00:21:39.639>this <00:21:39.733>model <00:21:39.826>behaves, <00:21:39.919>what <00:21:40.013>do <00:21:40.106>I <00:21:40.200>need

00:21:40.870 --> 00:21:40.880 align:start position:0%
<00:21:40.870>how <00:21:40.871>this <00:21:40.872>model <00:21:40.873>behaves, <00:21:40.875>what <00:21:40.876>do <00:21:40.877>I <00:21:40.879>need

00:21:40.880 --> 00:21:42.390 align:start position:0%
<00:21:40.880>how <00:21:40.970>this <00:21:41.061>model <00:21:41.151>behaves, <00:21:41.242>what <00:21:41.333>do <00:21:41.423>I <00:21:41.514>need <00:21:41.604>to <00:21:41.695>change <00:21:41.786>in <00:21:41.876>the <00:21:41.967>system <00:21:42.057>prompt, <00:21:42.148>what <00:21:42.239>do

00:21:42.390 --> 00:21:42.400 align:start position:0%
<00:21:42.390>to <00:21:42.391>change <00:21:42.392>in <00:21:42.393>the <00:21:42.395>system <00:21:42.396>prompt, <00:21:42.397>what <00:21:42.399>do

00:21:42.400 --> 00:21:44.350 align:start position:0%
<00:21:42.400>to <00:21:42.493>change <00:21:42.586>in <00:21:42.680>the <00:21:42.773>system <00:21:42.866>prompt, <00:21:42.960>what <00:21:43.053>do <00:21:43.146>I <00:21:43.240>need <00:21:43.333>to <00:21:43.426>change <00:21:43.520>in <00:21:43.613>the <00:21:43.706>tool <00:21:43.800>calling.

00:21:44.350 --> 00:21:44.360 align:start position:0%
<00:21:44.350>I <00:21:44.351>need <00:21:44.352>to <00:21:44.353>change <00:21:44.355>in <00:21:44.356>the <00:21:44.357>tool <00:21:44.359>calling.

00:21:44.360 --> 00:21:46.350 align:start position:0%
<00:21:44.360>I <00:21:44.453>need <00:21:44.546>to <00:21:44.639>change <00:21:44.733>in <00:21:44.826>the <00:21:44.919>tool <00:21:45.013>calling. <00:21:45.106>So, <00:21:45.199>it's <00:21:45.293>really <00:21:45.386>a <00:21:45.479>lot <00:21:45.573>of <00:21:45.666>currently <00:21:45.760>at

00:21:46.350 --> 00:21:46.360 align:start position:0%
<00:21:46.350>So, <00:21:46.351>it's <00:21:46.352>really <00:21:46.353>a <00:21:46.355>lot <00:21:46.356>of <00:21:46.357>currently <00:21:46.359>at

00:21:46.360 --> 00:21:49.190 align:start position:0%
<00:21:46.360>So, <00:21:46.459>it's <00:21:46.560>really <00:21:46.659>a <00:21:46.760>lot <00:21:46.860>of <00:21:46.959>currently <00:21:47.060>at <00:21:47.159>least <00:21:47.260>it's <00:21:47.360>a <00:21:47.459>lot <00:21:47.560>of <00:21:47.659>gut <00:21:47.760>feeling.

00:21:49.190 --> 00:21:49.200 align:start position:0%
<00:21:49.190>least <00:21:49.191>it's <00:21:49.193>a <00:21:49.194>lot <00:21:49.196>of <00:21:49.197>gut <00:21:49.199>feeling.

00:21:49.200 --> 00:21:51.950 align:start position:0%
<00:21:49.200>least <00:21:49.293>it's <00:21:49.386>a <00:21:49.480>lot <00:21:49.573>of <00:21:49.666>gut <00:21:49.760>feeling. <00:21:49.853>&gt;&gt; <00:21:49.946>How <00:21:50.040>do <00:21:50.133>you <00:21:50.226>ensure <00:21:50.320>or <00:21:50.413>test <00:21:50.506>that <00:21:50.600>updating

00:21:51.950 --> 00:21:51.960 align:start position:0%
<00:21:51.950>&gt;&gt; <00:21:51.951>How <00:21:51.952>do <00:21:51.953>you <00:21:51.954>ensure <00:21:51.955>or <00:21:51.956>test <00:21:51.957>that <00:21:51.959>updating

00:21:51.960 --> 00:21:53.750 align:start position:0%
<00:21:51.960>&gt;&gt; <00:21:52.042>How <00:21:52.124>do <00:21:52.207>you <00:21:52.289>ensure <00:21:52.371>or <00:21:52.454>test <00:21:52.536>that <00:21:52.618>updating <00:21:52.701>all <00:21:52.783>it <00:21:52.865>to <00:21:52.948>a <00:21:53.030>new <00:21:53.112>model <00:21:53.195>will <00:21:53.277>make <00:21:53.360>it

00:21:53.750 --> 00:21:53.760 align:start position:0%
<00:21:53.750>all <00:21:53.751>it <00:21:53.752>to <00:21:53.753>a <00:21:53.754>new <00:21:53.755>model <00:21:53.756>will <00:21:53.757>make <00:21:53.759>it

00:21:53.760 --> 00:21:56.230 align:start position:0%
<00:21:53.760>all <00:21:53.847>it <00:21:53.935>to <00:21:54.022>a <00:21:54.110>new <00:21:54.197>model <00:21:54.285>will <00:21:54.372>make <00:21:54.460>it <00:21:54.547>better <00:21:54.635>or <00:21:54.722>and <00:21:54.810>not <00:21:54.897>worse? <00:21:54.985>Do <00:21:55.072>you <00:21:55.160>only

00:21:56.230 --> 00:21:56.240 align:start position:0%
<00:21:56.230>better <00:21:56.231>or <00:21:56.232>and <00:21:56.233>not <00:21:56.235>worse? <00:21:56.236>Do <00:21:56.237>you <00:21:56.239>only

00:21:56.240 --> 00:21:58.230 align:start position:0%
<00:21:56.240>better <00:21:56.347>or <00:21:56.455>and <00:21:56.563>not <00:21:56.670>worse? <00:21:56.778>Do <00:21:56.886>you <00:21:56.993>only <00:21:57.101>check <00:21:57.209>this <00:21:57.316>in <00:21:57.424>production <00:21:57.532>through <00:21:57.640>the

00:21:58.230 --> 00:21:58.240 align:start position:0%
<00:21:58.230>check <00:21:58.231>this <00:21:58.233>in <00:21:58.235>production <00:21:58.237>through <00:21:58.239>the

00:21:58.240 --> 00:22:00.790 align:start position:0%
<00:21:58.240>check <00:21:58.347>this <00:21:58.455>in <00:21:58.563>production <00:21:58.670>through <00:21:58.778>the <00:21:58.886>canary <00:21:58.993>release <00:21:59.101>or <00:21:59.209>do <00:21:59.316>you <00:21:59.424>have <00:21:59.532>test <00:21:59.640>for

00:22:00.790 --> 00:22:00.800 align:start position:0%
<00:22:00.790>canary <00:22:00.791>release <00:22:00.792>or <00:22:00.793>do <00:22:00.795>you <00:22:00.796>have <00:22:00.797>test <00:22:00.799>for

00:22:00.800 --> 00:22:02.270 align:start position:0%
<00:22:00.800>canary <00:22:00.947>release <00:22:01.094>or <00:22:01.241>do <00:22:01.388>you <00:22:01.534>have <00:22:01.682>test <00:22:01.829>for <00:22:01.976>your <00:22:02.123>prompts?

00:22:02.270 --> 00:22:02.280 align:start position:0%
<00:22:02.270>your <00:22:02.279>prompts?

00:22:02.280 --> 00:22:04.830 align:start position:0%
<00:22:02.280>your <00:22:02.435>prompts? <00:22:02.591>&gt;&gt; <00:22:02.746>No, <00:22:02.902>so <00:22:03.057>obviously <00:22:03.213>not <00:22:03.368>only <00:22:03.524>in <00:22:03.680>production.

00:22:04.830 --> 00:22:04.840 align:start position:0%
<00:22:04.830>&gt;&gt; <00:22:04.831>No, <00:22:04.832>so <00:22:04.833>obviously <00:22:04.835>not <00:22:04.836>only <00:22:04.837>in <00:22:04.839>production.

00:22:04.840 --> 00:22:07.270 align:start position:0%
<00:22:04.840>&gt;&gt; <00:22:04.947>No, <00:22:05.055>so <00:22:05.163>obviously <00:22:05.270>not <00:22:05.378>only <00:22:05.486>in <00:22:05.593>production. <00:22:05.701>We <00:22:05.809>We <00:22:05.916>are <00:22:06.024>constantly <00:22:06.132>testing <00:22:06.240>before

00:22:07.270 --> 00:22:07.280 align:start position:0%
<00:22:07.270>We <00:22:07.271>We <00:22:07.273>are <00:22:07.275>constantly <00:22:07.277>testing <00:22:07.279>before

00:22:07.280 --> 00:22:09.870 align:start position:0%
<00:22:07.280>We <00:22:07.387>We <00:22:07.495>are <00:22:07.603>constantly <00:22:07.710>testing <00:22:07.818>before <00:22:07.926>production <00:22:08.033>and <00:22:08.141>this <00:22:08.249>is <00:22:08.356>the <00:22:08.464>evil <00:22:08.572>system <00:22:08.680>I

00:22:09.870 --> 00:22:09.880 align:start position:0%
<00:22:09.870>production <00:22:09.871>and <00:22:09.872>this <00:22:09.873>is <00:22:09.875>the <00:22:09.876>evil <00:22:09.877>system <00:22:09.879>I

00:22:09.880 --> 00:22:12.230 align:start position:0%
<00:22:09.880>production <00:22:09.980>and <00:22:10.080>this <00:22:10.180>is <00:22:10.280>the <00:22:10.380>evil <00:22:10.480>system <00:22:10.580>I <00:22:10.680>was <00:22:10.780>talking <00:22:10.880>about. <00:22:10.980>And <00:22:11.080>we <00:22:11.180>are <00:22:11.280>constantly

00:22:12.230 --> 00:22:12.240 align:start position:0%
<00:22:12.230>was <00:22:12.231>talking <00:22:12.233>about. <00:22:12.234>And <00:22:12.236>we <00:22:12.237>are <00:22:12.239>constantly

00:22:12.240 --> 00:22:13.910 align:start position:0%
<00:22:12.240>was <00:22:12.347>talking <00:22:12.455>about. <00:22:12.563>And <00:22:12.670>we <00:22:12.778>are <00:22:12.886>constantly <00:22:12.993>by <00:22:13.101>the <00:22:13.209>way <00:22:13.316>adding <00:22:13.424>more <00:22:13.532>and <00:22:13.640>more

00:22:13.910 --> 00:22:13.920 align:start position:0%
<00:22:13.910>by <00:22:13.911>the <00:22:13.913>way <00:22:13.914>adding <00:22:13.916>more <00:22:13.917>and <00:22:13.919>more

00:22:13.920 --> 00:22:16.470 align:start position:0%
<00:22:13.920>by <00:22:14.027>the <00:22:14.135>way <00:22:14.243>adding <00:22:14.350>more <00:22:14.458>and <00:22:14.566>more <00:22:14.673>evaluation <00:22:14.781>test <00:22:14.889>cases. <00:22:14.996>So, <00:22:15.104>we <00:22:15.212>want <00:22:15.320>to

00:22:16.470 --> 00:22:16.480 align:start position:0%
<00:22:16.470>evaluation <00:22:16.471>test <00:22:16.473>cases. <00:22:16.474>So, <00:22:16.476>we <00:22:16.477>want <00:22:16.479>to

00:22:16.480 --> 00:22:19.390 align:start position:0%
<00:22:16.480>evaluation <00:22:16.580>test <00:22:16.680>cases. <00:22:16.780>So, <00:22:16.880>we <00:22:16.980>want <00:22:17.080>to <00:22:17.180>see <00:22:17.280>that <00:22:17.380>all <00:22:17.480>the <00:22:17.580>increases <00:22:17.680>in <00:22:17.780>score <00:22:17.880>and

00:22:19.390 --> 00:22:19.400 align:start position:0%
<00:22:19.390>see <00:22:19.391>that <00:22:19.392>all <00:22:19.393>the <00:22:19.395>increases <00:22:19.396>in <00:22:19.397>score <00:22:19.399>and

00:22:19.400 --> 00:22:21.750 align:start position:0%
<00:22:19.400>see <00:22:19.507>that <00:22:19.615>all <00:22:19.723>the <00:22:19.830>increases <00:22:19.938>in <00:22:20.046>score <00:22:20.153>and <00:22:20.261>also <00:22:20.369>success <00:22:20.476>rate <00:22:20.584>is <00:22:20.692>increasing <00:22:20.800>over

00:22:21.750 --> 00:22:21.760 align:start position:0%
<00:22:21.750>also <00:22:21.751>success <00:22:21.753>rate <00:22:21.755>is <00:22:21.757>increasing <00:22:21.759>over

00:22:21.760 --> 00:22:22.590 align:start position:0%
<00:22:21.760>also <00:22:21.884>success <00:22:22.009>rate <00:22:22.133>is <00:22:22.258>increasing <00:22:22.382>over <00:22:22.506>time.

00:22:22.590 --> 00:22:22.600 align:start position:0%
time.

00:22:22.600 --> 00:22:22.990 align:start position:0%
<00:22:22.600>time. <00:22:22.775>&gt;&gt; <00:22:22.951>And

00:22:22.990 --> 00:22:23.000 align:start position:0%
<00:22:22.990>&gt;&gt; <00:22:22.999>And

00:22:23.000 --> 00:22:24.670 align:start position:0%
<00:22:23.000>&gt;&gt; <00:22:23.155>And <00:22:23.311>&gt;&gt; <00:22:23.466>And <00:22:23.622>we <00:22:23.777>won't <00:22:23.933>release <00:22:24.088>it <00:22:24.244>to <00:22:24.400>production

00:22:24.670 --> 00:22:24.680 align:start position:0%
<00:22:24.670>&gt;&gt; <00:22:24.671>And <00:22:24.672>we <00:22:24.673>won't <00:22:24.675>release <00:22:24.676>it <00:22:24.677>to <00:22:24.679>production

00:22:24.680 --> 00:22:26.830 align:start position:0%
<00:22:24.680>&gt;&gt; <00:22:24.787>And <00:22:24.895>we <00:22:25.003>won't <00:22:25.110>release <00:22:25.218>it <00:22:25.326>to <00:22:25.433>production <00:22:25.541>unless <00:22:25.649>we <00:22:25.756>we <00:22:25.864>meet <00:22:25.972>these <00:22:26.080>goals.

00:22:26.830 --> 00:22:26.840 align:start position:0%
<00:22:26.830>unless <00:22:26.831>we <00:22:26.833>we <00:22:26.835>meet <00:22:26.837>these <00:22:26.839>goals.

00:22:26.840 --> 00:22:27.750 align:start position:0%
<00:22:26.840>unless <00:22:26.956>we <00:22:27.073>we <00:22:27.191>meet <00:22:27.308>these <00:22:27.425>goals. <00:22:27.542>&gt;&gt; <00:22:27.659>Yeah.

00:22:27.750 --> 00:22:27.760 align:start position:0%
<00:22:27.750>&gt;&gt; <00:22:27.759>Yeah.

00:22:27.760 --> 00:22:29.710 align:start position:0%
<00:22:27.760>&gt;&gt; <00:22:27.915>Yeah. <00:22:28.071>How <00:22:28.226>do <00:22:28.382>you <00:22:28.537>migrate <00:22:28.693>safely <00:22:28.848>from <00:22:29.004>one <00:22:29.160>model

00:22:29.710 --> 00:22:29.720 align:start position:0%
<00:22:29.710>How <00:22:29.711>do <00:22:29.712>you <00:22:29.713>migrate <00:22:29.715>safely <00:22:29.716>from <00:22:29.717>one <00:22:29.719>model

00:22:29.720 --> 00:22:31.510 align:start position:0%
<00:22:29.720>How <00:22:29.813>do <00:22:29.906>you <00:22:30.000>migrate <00:22:30.093>safely <00:22:30.186>from <00:22:30.280>one <00:22:30.373>model <00:22:30.466>to <00:22:30.560>another? <00:22:30.653>Do <00:22:30.746>you <00:22:30.840>have <00:22:30.933>to <00:22:31.026>rewrite <00:22:31.120>the

00:22:31.510 --> 00:22:31.520 align:start position:0%
<00:22:31.510>to <00:22:31.511>another? <00:22:31.512>Do <00:22:31.513>you <00:22:31.515>have <00:22:31.516>to <00:22:31.517>rewrite <00:22:31.519>the

00:22:31.520 --> 00:22:32.590 align:start position:0%
<00:22:31.520>to <00:22:31.627>another? <00:22:31.734>Do <00:22:31.841>you <00:22:31.947>have <00:22:32.054>to <00:22:32.162>rewrite <00:22:32.269>the <00:22:32.376>whole <00:22:32.483>prompt?

00:22:32.590 --> 00:22:32.600 align:start position:0%
<00:22:32.590>whole <00:22:32.599>prompt?

00:22:32.600 --> 00:22:33.390 align:start position:0%
<00:22:32.600>whole <00:22:32.718>prompt? <00:22:32.837>&gt;&gt; <00:22:32.955>Sorry, <00:22:33.074>say <00:22:33.192>it <00:22:33.311>again.

00:22:33.390 --> 00:22:33.400 align:start position:0%
<00:22:33.390>&gt;&gt; <00:22:33.392>Sorry, <00:22:33.394>say <00:22:33.396>it <00:22:33.399>again.

00:22:33.400 --> 00:22:34.950 align:start position:0%
<00:22:33.400>&gt;&gt; <00:22:33.516>Sorry, <00:22:33.632>say <00:22:33.748>it <00:22:33.865>again. <00:22:33.981>&gt;&gt; <00:22:34.097>How <00:22:34.213>to <00:22:34.330>migrate <00:22:34.446>from <00:22:34.562>one <00:22:34.678>model <00:22:34.795>to

00:22:34.950 --> 00:22:34.960 align:start position:0%
<00:22:34.950>&gt;&gt; <00:22:34.951>How <00:22:34.952>to <00:22:34.953>migrate <00:22:34.955>from <00:22:34.956>one <00:22:34.957>model <00:22:34.959>to

00:22:34.960 --> 00:22:36.790 align:start position:0%
<00:22:34.960>&gt;&gt; <00:22:35.060>How <00:22:35.160>to <00:22:35.260>migrate <00:22:35.360>from <00:22:35.460>one <00:22:35.560>model <00:22:35.660>to <00:22:35.760>another? <00:22:35.860>Do <00:22:35.960>you <00:22:36.060>have <00:22:36.160>to <00:22:36.260>rewrite <00:22:36.360>the

00:22:36.790 --> 00:22:36.800 align:start position:0%
<00:22:36.790>another? <00:22:36.791>Do <00:22:36.793>you <00:22:36.794>have <00:22:36.796>to <00:22:36.797>rewrite <00:22:36.799>the

00:22:36.800 --> 00:22:38.950 align:start position:0%
<00:22:36.800>another? <00:22:36.975>Do <00:22:37.149>you <00:22:37.325>have <00:22:37.500>to <00:22:37.675>rewrite <00:22:37.850>the <00:22:38.024>whole <00:22:38.200>prompt?

00:22:38.950 --> 00:22:38.960 align:start position:0%
<00:22:38.950>whole <00:22:38.959>prompt?

00:22:38.960 --> 00:22:41.030 align:start position:0%
<00:22:38.960>whole <00:22:39.193>prompt? <00:22:39.426>&gt;&gt; <00:22:39.660>It's <00:22:39.893>a <00:22:40.126>good <00:22:40.360>question.

00:22:41.030 --> 00:22:41.040 align:start position:0%
<00:22:41.030>&gt;&gt; <00:22:41.032>It's <00:22:41.034>a <00:22:41.036>good <00:22:41.039>question.

00:22:41.040 --> 00:22:43.150 align:start position:0%
<00:22:41.040>&gt;&gt; <00:22:41.180>It's <00:22:41.320>a <00:22:41.460>good <00:22:41.600>question. <00:22:41.740>Usually <00:22:41.879>it's <00:22:42.020>not <00:22:42.159>rewriting <00:22:42.300>the <00:22:42.440>whole

00:22:43.150 --> 00:22:43.160 align:start position:0%
<00:22:43.150>Usually <00:22:43.151>it's <00:22:43.153>not <00:22:43.155>rewriting <00:22:43.157>the <00:22:43.159>whole

00:22:43.160 --> 00:22:44.710 align:start position:0%
<00:22:43.160>Usually <00:22:43.267>it's <00:22:43.374>not <00:22:43.481>rewriting <00:22:43.589>the <00:22:43.696>whole <00:22:43.803>prompt. <00:22:43.911>I <00:22:44.018>will <00:22:44.125>say <00:22:44.233>that <00:22:44.340>with <00:22:44.447>when <00:22:44.555>you

00:22:44.710 --> 00:22:44.720 align:start position:0%
<00:22:44.710>prompt. <00:22:44.711>I <00:22:44.712>will <00:22:44.713>say <00:22:44.715>that <00:22:44.716>with <00:22:44.717>when <00:22:44.719>you

00:22:44.720 --> 00:22:46.470 align:start position:0%
<00:22:44.720>prompt. <00:22:44.820>I <00:22:44.920>will <00:22:45.020>say <00:22:45.120>that <00:22:45.220>with <00:22:45.320>when <00:22:45.420>you <00:22:45.520>move <00:22:45.620>from <00:22:45.720>one <00:22:45.820>family, <00:22:45.920>so <00:22:46.020>for <00:22:46.120>example

00:22:46.470 --> 00:22:46.480 align:start position:0%
<00:22:46.470>move <00:22:46.471>from <00:22:46.473>one <00:22:46.474>family, <00:22:46.476>so <00:22:46.477>for <00:22:46.479>example

00:22:46.480 --> 00:22:49.790 align:start position:0%
<00:22:46.480>move <00:22:46.587>from <00:22:46.695>one <00:22:46.803>family, <00:22:46.910>so <00:22:47.018>for <00:22:47.126>example <00:22:47.233>from <00:22:47.341>cloud <00:22:47.449>to <00:22:47.556>GPT <00:22:47.664>or <00:22:47.772>to <00:22:47.880>Gemini,

00:22:49.790 --> 00:22:49.800 align:start position:0%
<00:22:49.790>from <00:22:49.791>cloud <00:22:49.793>to <00:22:49.794>GPT <00:22:49.796>or <00:22:49.797>to <00:22:49.799>Gemini,

00:22:49.800 --> 00:22:51.910 align:start position:0%
<00:22:49.800>from <00:22:49.899>cloud <00:22:50.000>to <00:22:50.100>GPT <00:22:50.200>or <00:22:50.300>to <00:22:50.399>Gemini, <00:22:50.500>yes, <00:22:50.600>it <00:22:50.700>is <00:22:50.800>recommended <00:22:50.899>to <00:22:51.000>change <00:22:51.100>a <00:22:51.200>lot

00:22:51.910 --> 00:22:51.920 align:start position:0%
<00:22:51.910>yes, <00:22:51.911>it <00:22:51.912>is <00:22:51.913>recommended <00:22:51.915>to <00:22:51.916>change <00:22:51.917>a <00:22:51.919>lot

00:22:51.920 --> 00:22:53.270 align:start position:0%
<00:22:51.920>yes, <00:22:52.001>it <00:22:52.082>is <00:22:52.163>recommended <00:22:52.244>to <00:22:52.325>change <00:22:52.406>a <00:22:52.487>lot <00:22:52.568>of <00:22:52.649>things <00:22:52.730>in <00:22:52.811>the <00:22:52.892>system <00:22:52.973>prompt. <00:22:53.054>So, <00:22:53.135>it's

00:22:53.270 --> 00:22:53.280 align:start position:0%
<00:22:53.270>of <00:22:53.271>things <00:22:53.272>in <00:22:53.273>the <00:22:53.275>system <00:22:53.276>prompt. <00:22:53.277>So, <00:22:53.279>it's

00:22:53.280 --> 00:22:54.630 align:start position:0%
<00:22:53.280>of <00:22:53.366>things <00:22:53.453>in <00:22:53.540>the <00:22:53.627>system <00:22:53.713>prompt. <00:22:53.800>So, <00:22:53.887>it's <00:22:53.974>almost <00:22:54.061>a <00:22:54.147>different <00:22:54.234>prompt. <00:22:54.321>So, <00:22:54.408>we <00:22:54.495>have

00:22:54.630 --> 00:22:54.640 align:start position:0%
<00:22:54.630>almost <00:22:54.631>a <00:22:54.633>different <00:22:54.634>prompt. <00:22:54.636>So, <00:22:54.637>we <00:22:54.639>have

00:22:54.640 --> 00:22:55.790 align:start position:0%
<00:22:54.640>almost <00:22:54.769>a <00:22:54.898>different <00:22:55.028>prompt. <00:22:55.157>So, <00:22:55.286>we <00:22:55.416>have <00:22:55.545>like <00:22:55.675>a

00:22:55.790 --> 00:22:55.800 align:start position:0%
<00:22:55.790>like <00:22:55.799>a

00:22:55.800 --> 00:22:58.310 align:start position:0%
<00:22:55.800>like <00:22:56.000>a <00:22:56.200>separate <00:22:56.399>prompt <00:22:56.600>system <00:22:56.800>prompts <00:22:57.000>for <00:22:57.200>every

00:22:58.310 --> 00:22:58.320 align:start position:0%
<00:22:58.310>separate <00:22:58.311>prompt <00:22:58.313>system <00:22:58.315>prompts <00:22:58.317>for <00:22:58.319>every

00:22:58.320 --> 00:22:59.910 align:start position:0%
<00:22:58.320>separate <00:22:58.520>prompt <00:22:58.720>system <00:22:58.919>prompts <00:22:59.120>for <00:22:59.320>every <00:22:59.520>model <00:22:59.720>family.

00:22:59.910 --> 00:22:59.920 align:start position:0%
<00:22:59.910>model <00:22:59.919>family.

00:22:59.920 --> 00:23:01.830 align:start position:0%
<00:22:59.920>model <00:23:00.060>family. <00:23:00.200>&gt;&gt; <00:23:00.340>I <00:23:00.480>see. <00:23:00.620>Thank <00:23:00.760>you. <00:23:00.900>Thank <00:23:01.040>you <00:23:01.180>very <00:23:01.320>much.

00:23:01.830 --> 00:23:01.840 align:start position:0%
<00:23:01.830>&gt;&gt; <00:23:01.831>I <00:23:01.832>see. <00:23:01.833>Thank <00:23:01.834>you. <00:23:01.835>Thank <00:23:01.836>you <00:23:01.837>very <00:23:01.839>much.

00:23:01.840 --> 00:23:02.710 align:start position:0%
<00:23:01.840>&gt;&gt; <00:23:01.918>I <00:23:01.996>see. <00:23:02.074>Thank <00:23:02.153>you. <00:23:02.231>Thank <00:23:02.309>you <00:23:02.388>very <00:23:02.466>much. <00:23:02.544>Thank <00:23:02.623>you.

00:23:02.710 --> 00:23:02.720 align:start position:0%
<00:23:02.710>Thank <00:23:02.719>you.

00:23:02.720 --> 00:23:04.740 align:start position:0%
<00:23:02.720>Thank <00:23:03.186>you. <00:23:03.653>&gt;&gt; <00:23:04.120>[applause]