WEBVTT

1
00:00:05.005 --> 00:00:06.135
<00:00:05.005>Welcome <00:00:05.343>to <00:00:05.683>my <00:00:06.022>talk.

2
00:00:06.315 --> 00:00:08.935
<00:00:06.315>UN <00:00:06.515>reeling <00:00:06.715>the <00:00:06.915>Magic <00:00:07.115>Scaling <00:00:07.315>Large <00:00:07.515>ranges <00:00:07.715>models

3
00:00:09.355 --> 00:00:10.415
<00:00:09.355>to <00:00:09.832>serve <00:00:10.309>Millions.

4
00:00:10.715 --> 00:00:12.255
<00:00:10.715>It <00:00:10.992>sounds <00:00:11.269>like <00:00:11.546>a <00:00:11.823>clickbait <00:00:12.101>title,

5
00:00:12.835 --> 00:00:14.935
<00:00:12.835>but <00:00:13.068>there's <00:00:13.301>a <00:00:13.535>lot <00:00:13.768>of <00:00:14.001>magic <00:00:14.235>involved,

6
00:00:15.355 --> 00:00:17.015
<00:00:15.355>so <00:00:15.555>let's <00:00:15.755>figure <00:00:15.955>it <00:00:16.155>out <00:00:16.355>what <00:00:16.555>it <00:00:16.755>is.

7
00:00:17.475 --> 00:00:20.655
<00:00:17.475>In <00:00:17.602>a <00:00:17.729>couple <00:00:17.856>of <00:00:17.984>last <00:00:18.111>sessions, <00:00:18.238>we <00:00:18.365>saw <00:00:18.493>a <00:00:18.620>lot <00:00:18.747>of <00:00:18.875>AI

8
00:00:20.915 --> 00:00:25.575
<00:00:20.915>and <00:00:21.148>staff, <00:00:21.381>H <00:00:21.615>ai, <00:00:21.848>AI <00:00:22.081>agent <00:00:22.314>systems.

9
00:00:26.605 --> 00:00:29.175
<00:00:26.605>They're <00:00:26.838>all <00:00:27.071>using <00:00:27.305>this <00:00:27.538>broad <00:00:27.771>engineering <00:00:28.005>tricks,

10
00:00:29.315 --> 00:00:31.975
<00:00:29.315>the <00:00:29.595>context <00:00:29.875>engineering <00:00:30.155>tricks, <00:00:30.435>tool <00:00:30.715>callings,

11
00:00:32.105 --> 00:00:34.695
<00:00:32.104>short <00:00:32.455>term <00:00:32.805>memory, <00:00:33.154>long-term <00:00:33.504>memory,

12
00:00:34.995 --> 00:00:38.735
<00:00:34.995>but <00:00:35.150>at <00:00:35.306>the <00:00:35.461>heart <00:00:35.617>of <00:00:35.772>any <00:00:35.928>agent <00:00:36.083>system's, <00:00:36.239>the <00:00:36.394>LM.

13
00:00:38.735 --> 00:00:40.495
<00:00:38.735>So <00:00:39.015>we <00:00:39.295>are <00:00:39.575>talking <00:00:39.855>about <00:00:40.135>LLMs.

14
00:00:41.645 --> 00:00:44.175
<00:00:41.645>Most <00:00:41.820>companies <00:00:41.995>are <00:00:42.170>probably <00:00:42.345>using <00:00:42.520>the <00:00:42.695>frontier <00:00:42.870>models <00:00:43.045>from

15
00:00:44.395 --> 00:00:46.655
<00:00:44.395>OpenAI <00:00:44.745>and <00:00:45.095>Tropic <00:00:45.445>or <00:00:45.795>Google,

16
00:00:47.075 --> 00:00:49.175
<00:00:47.075>but <00:00:47.308>there's <00:00:47.541>a <00:00:47.775>equal <00:00:48.008>amount <00:00:48.241>of <00:00:48.475>companies

17
00:00:49.175 --> 00:00:51.295
<00:00:49.175>that <00:00:49.525>are <00:00:49.875>concerned <00:00:50.224>about <00:00:50.574>this.

18
00:00:52.215 --> 00:00:54.015
<00:00:52.215>Frontier <00:00:52.448>models <00:00:52.681>and <00:00:52.915>host <00:00:53.148>their <00:00:53.381>own <00:00:53.615>large

19
00:00:54.255 --> 00:00:55.375
<00:00:54.255>language <00:00:54.456>models <00:00:54.658>in <00:00:54.859>the <00:00:55.061>data <00:00:55.263>center.

20
00:00:56.355 --> 00:00:59.935
<00:00:56.355>And <00:00:56.495>so <00:00:56.635>do <00:00:56.775>we <00:00:56.915>at <00:00:57.055>Stack. <00:00:57.194>It's <00:00:57.334>a <00:00:57.474>European <00:00:57.614>cloud <00:00:57.754>provider.

21
00:01:00.275 --> 00:01:03.175
<00:01:00.275>In <00:01:00.415>the <00:01:00.555>last <00:01:00.695>year <00:01:00.835>we <00:01:00.975>have <00:01:01.114>built <00:01:01.254>a <00:01:01.394>model <00:01:01.535>sew <00:01:01.675>service

22
00:01:03.835 --> 00:01:06.295
<00:01:03.835>and <00:01:04.035>we <00:01:04.235>started <00:01:04.435>with <00:01:04.635>serving <00:01:04.834>large <00:01:05.035>language <00:01:05.235>models,

23
00:01:06.475 --> 00:01:11.055
<00:01:06.475>but <00:01:06.708>it's <00:01:06.941>not <00:01:07.175>only, <00:01:07.408>um, <00:01:07.641>not <00:01:07.875>only,

24
00:01:11.435 --> 00:01:13.895
<00:01:11.435>uh, <00:01:11.715>restricted <00:01:11.995>to <00:01:12.275>large <00:01:12.555>language <00:01:12.835>models,

25
00:01:13.895 --> 00:01:16.295
<00:01:13.895>but <00:01:14.095>can <00:01:14.295>be <00:01:14.494>used <00:01:14.695>for <00:01:14.895>machine <00:01:15.095>learning <00:01:15.295>models,

26
00:01:17.095 --> 00:01:18.735
<00:01:17.095>models, <00:01:17.561>deep <00:01:18.028>learning <00:01:18.495>models.

27
00:01:19.635 --> 00:01:21.815
<00:01:19.635>So <00:01:19.868>they're <00:01:20.101>all <00:01:20.335>following <00:01:20.568>the <00:01:20.801>same <00:01:21.035>principle.

28
00:01:22.875 --> 00:01:24.935
<00:01:22.875>I'm <00:01:23.108>looking, <00:01:23.341>uh, <00:01:23.575>through <00:01:23.808>the <00:01:24.041>software <00:01:24.275>engineering

29
00:01:24.935 --> 00:01:26.495
<00:01:24.935>lens <00:01:25.401>in <00:01:25.868>an <00:01:26.335>LLM.

30
00:01:26.515 --> 00:01:29.255
<00:01:26.515>So <00:01:26.670>I'm <00:01:26.826>not <00:01:26.981>an <00:01:27.137>AI <00:01:27.292>engineer, <00:01:27.448>I <00:01:27.603>don't <00:01:27.759>really <00:01:27.915>care,

31
00:01:29.255 --> 00:01:30.455
<00:01:29.255>okay, <00:01:29.470>how <00:01:29.687>can <00:01:29.902>we <00:01:30.119>train <00:01:30.335>it?

32
00:01:30.555 --> 00:01:34.295
<00:01:30.555>How <00:01:30.755>can <00:01:30.955>we <00:01:31.155>optimize <00:01:31.355>how <00:01:31.555>the <00:01:31.755>LLM <00:01:31.955>behaves?

33
00:01:34.835 --> 00:01:37.055
<00:01:34.835>I'm <00:01:35.010>more <00:01:35.185>interested, <00:01:35.360>how <00:01:35.535>can <00:01:35.710>we <00:01:35.885>make <00:01:36.060>it <00:01:36.235>reliable,

34
00:01:37.395 --> 00:01:40.175
<00:01:37.395>how <00:01:37.628>we <00:01:37.861>can <00:01:38.095>save, <00:01:38.328>serve <00:01:38.561>more <00:01:38.795>requests,

35
00:01:41.035 --> 00:01:42.735
<00:01:41.035>how <00:01:41.315>we <00:01:41.595>can <00:01:41.875>optimize <00:01:42.155>the <00:01:42.435>latency.

36
00:01:43.315 --> 00:01:46.135
<00:01:43.315>And <00:01:43.455>so <00:01:43.595>this <00:01:43.735>talk <00:01:43.875>is <00:01:44.015>rather <00:01:44.155>going <00:01:44.295>into <00:01:44.435>the <00:01:44.575>direction <00:01:44.715>of

37
00:01:47.275 --> 00:01:48.695
<00:01:47.275>the <00:01:47.701>software <00:01:48.127>engineering <00:01:48.553>lens.

38
00:01:48.955 --> 00:01:50.855
<00:01:48.955>So <00:01:49.421>first, <00:01:49.888>the <00:01:50.355>introduction

39
00:01:50.875 --> 00:01:52.655
<00:01:50.875>by <00:01:51.075>you <00:01:51.275>should <00:01:51.475>ever <00:01:51.675>host <00:01:51.875>your <00:01:52.075>own <00:01:52.275>airlines.

40
00:01:52.715 --> 00:01:54.695
<00:01:52.715>Why <00:01:53.065>don't <00:01:53.415>simply <00:01:53.765>use <00:01:54.115>OpenAI?

41
00:01:56.195 --> 00:01:59.815
<00:01:56.195>The <00:01:56.350>big <00:01:56.506>main <00:01:56.661>selling <00:01:56.817>point <00:01:56.972>is <00:01:57.128>data <00:01:57.283>privacy, <00:01:57.439>of <00:01:57.595>course.

42
00:02:00.025 --> 00:02:02.495
<00:02:00.025>Maybe <00:02:00.225>you <00:02:00.425>have <00:02:00.625>company <00:02:00.825>data <00:02:01.025>like <00:02:01.225>some <00:02:01.425>customer

43
00:02:02.815 --> 00:02:04.055
<00:02:02.815>contracts <00:02:03.187>or <00:02:03.559>employee <00:02:03.931>data.

44
00:02:04.405 --> 00:02:06.815
<00:02:04.405>There's <00:02:04.560>all <00:02:04.716>sorts <00:02:04.871>of <00:02:05.027>data <00:02:05.182>you <00:02:05.338>probably <00:02:05.493>don't <00:02:05.649>wanna <00:02:05.805>leak

45
00:02:06.815 --> 00:02:10.815
<00:02:06.815>as <00:02:06.970>a <00:02:07.126>business <00:02:07.281>because <00:02:07.437>maybe <00:02:07.592>some <00:02:07.748>competitor <00:02:07.903>will <00:02:08.059>copy <00:02:08.215>your

46
00:02:10.865 --> 00:02:13.735
<00:02:10.865>ideas <00:02:11.215>and <00:02:11.565>you <00:02:11.915>never <00:02:12.265>know

47
00:02:13.735 --> 00:02:15.695
<00:02:13.735>what <00:02:13.968>the <00:02:14.201>providers <00:02:14.435>are <00:02:14.668>doing <00:02:14.901>at <00:02:15.135>OpenAI.

48
00:02:15.755 --> 00:02:17.855
<00:02:15.755>Do <00:02:15.930>they <00:02:16.105>save <00:02:16.280>the <00:02:16.454>data? <00:02:16.630>Do <00:02:16.805>they <00:02:16.980>lock <00:02:17.155>the

49
00:02:17.855 --> 00:02:18.935
<00:02:17.855>data <00:02:18.179>of <00:02:18.503>your <00:02:18.827>problems?

50
00:02:19.395 --> 00:02:20.815
<00:02:19.395>Do <00:02:19.650>they <00:02:19.906>use <00:02:20.161>it <00:02:20.417>for <00:02:20.673>training?

51
00:02:21.115 --> 00:02:22.815
<00:02:21.115>Of <00:02:21.395>course <00:02:21.675>you <00:02:21.955>can <00:02:22.235>have <00:02:22.515>contracts,

52
00:02:23.355 --> 00:02:24.575
<00:02:23.355>but <00:02:23.574>you <00:02:23.794>still <00:02:24.013>don't <00:02:24.233>really <00:02:24.452>know

53
00:02:24.575 --> 00:02:26.095
<00:02:24.575>what <00:02:24.917>they're <00:02:25.259>doing <00:02:25.601>in <00:02:25.942>behind.

54
00:02:26.275 --> 00:02:30.335
<00:02:26.275>So <00:02:26.450>the <00:02:26.625>safest <00:02:26.800>way <00:02:26.975>is <00:02:27.150>to <00:02:27.325>host <00:02:27.500>Alan <00:02:27.675>yourself.

55
00:02:30.675 --> 00:02:32.935
<00:02:30.675>And <00:02:30.830>I <00:02:30.986>think <00:02:31.141>this <00:02:31.297>is <00:02:31.452>the <00:02:31.608>direction <00:02:31.763>where <00:02:31.919>a <00:02:32.075>lot

56
00:02:32.935 --> 00:02:36.695
<00:02:32.935>of <00:02:33.110>companies <00:02:33.285>going, <00:02:33.460>for <00:02:33.635>example, <00:02:33.810>Uber's <00:02:33.985>doing <00:02:34.160>it, <00:02:34.335>TikTok

57
00:02:36.875 --> 00:02:40.135
<00:02:36.875>or <00:02:37.075>Snowflake, <00:02:37.275>then <00:02:37.475>maybe <00:02:37.675>you <00:02:37.875>have <00:02:38.075>company <00:02:38.275>compliance.

58
00:02:40.565 --> 00:02:43.775
<00:02:40.565>Some <00:02:40.704>companies <00:02:40.845>says <00:02:40.984>okay, <00:02:41.125>you <00:02:41.265>are <00:02:41.405>not <00:02:41.545>allowed <00:02:41.685>to <00:02:41.825>use <00:02:41.965>AWS

59
00:02:43.775 --> 00:02:45.415
<00:02:43.775>where <00:02:44.008>some <00:02:44.241>of <00:02:44.475>the <00:02:44.708>models <00:02:44.941>are <00:02:45.175>hosted.

60
00:02:45.835 --> 00:02:48.175
<00:02:45.835>For <00:02:45.990>example, <00:02:46.146>on <00:02:46.301>bedrock <00:02:46.457>you <00:02:46.612>can <00:02:46.768>use <00:02:46.923>the <00:02:47.079>Glot <00:02:47.235>models

61
00:02:49.835 --> 00:02:51.015
<00:02:49.835>and <00:02:50.189>stuff <00:02:50.543>like <00:02:50.897>that.

62
00:02:51.415 --> 00:02:53.935
<00:02:51.415>Companies <00:02:51.648>get <00:02:51.881>really <00:02:52.114>creative <00:02:52.348>with <00:02:52.581>their <00:02:52.815>rules.

63
00:02:55.025 --> 00:02:57.175
<00:02:55.025>Maybe <00:02:55.180>at <00:02:55.336>some <00:02:55.491>point <00:02:55.647>you <00:02:55.802>have <00:02:55.958>trained <00:02:56.113>your <00:02:56.269>own <00:02:56.425>LM,

64
00:02:57.355 --> 00:03:00.685
<00:02:57.355>but <00:02:57.510>maybe <00:02:57.666>this <00:02:57.821>is <00:02:57.977>also <00:02:58.132>your <00:02:58.288>own <00:02:58.443>machine <00:02:58.599>learning <00:02:58.755>model,

65
00:03:00.685 --> 00:03:01.725
<00:03:00.685>your <00:03:00.919>own <00:03:01.153>deep <00:03:01.387>learning <00:03:01.620>model.

66
00:03:01.745 --> 00:03:04.605
<00:03:01.745>So <00:03:01.885>you <00:03:02.025>want <00:03:02.165>to <00:03:02.305>provide <00:03:02.445>it <00:03:02.585>to <00:03:02.725>a <00:03:02.865>end <00:03:03.005>user <00:03:03.145>somehow.

67
00:03:05.505 --> 00:03:07.685
<00:03:05.505>Um, <00:03:05.704>usually <00:03:05.905>it's <00:03:06.105>done <00:03:06.305>via <00:03:06.505>the <00:03:06.704>AI <00:03:06.905>engineers.

68
00:03:07.685 --> 00:03:09.805
<00:03:07.685>They <00:03:07.860>don't, <00:03:08.035>they <00:03:08.210>are <00:03:08.385>really <00:03:08.560>good <00:03:08.735>at <00:03:08.910>training <00:03:09.085>models,

69
00:03:09.865 --> 00:03:11.525
<00:03:09.865>but <00:03:10.098>they <00:03:10.331>don't <00:03:10.565>know <00:03:10.798>how <00:03:11.031>to <00:03:11.265>uh,

70
00:03:11.635 --> 00:03:13.205
<00:03:11.635>give <00:03:11.915>it <00:03:12.195>access <00:03:12.475>to <00:03:12.755>the <00:03:13.035>customer.

71
00:03:14.505 --> 00:03:16.755
<00:03:14.505>Cost <00:03:14.738>efficiency, <00:03:14.971>this <00:03:15.204>is <00:03:15.438>always <00:03:15.671>an <00:03:15.905>argument

72
00:03:16.815 --> 00:03:20.075
<00:03:16.815>of <00:03:17.015>do <00:03:17.215>it <00:03:17.415>yourself <00:03:17.615>versus <00:03:17.815>using <00:03:18.015>a <00:03:18.215>service

73
00:03:20.145 --> 00:03:21.675
<00:03:20.145>that <00:03:20.489>is <00:03:20.833>providing <00:03:21.177>the <00:03:21.522>functionality.

74
00:03:22.005 --> 00:03:23.795
<00:03:22.005>Doing <00:03:22.285>yourself <00:03:22.565>is <00:03:22.845>obviously, <00:03:23.125>uh, <00:03:23.405>cheaper

75
00:03:25.225 --> 00:03:27.515
<00:03:25.225>because <00:03:25.424>you <00:03:25.625>have <00:03:25.825>to <00:03:26.025>operate <00:03:26.225>it <00:03:26.424>as <00:03:26.625>well

76
00:03:27.655 --> 00:03:31.235
<00:03:27.655>and <00:03:27.830>all <00:03:28.005>the <00:03:28.180>costs <00:03:28.355>of <00:03:28.530>getting <00:03:28.705>the <00:03:28.880>revenue, <00:03:29.055>um,

77
00:03:31.335 --> 00:03:34.555
<00:03:31.335>for <00:03:31.490>the <00:03:31.646>employee <00:03:31.801>costs, <00:03:31.957>the <00:03:32.112>infrastructure <00:03:32.268>costs <00:03:32.423>and <00:03:32.579>so <00:03:32.735>on.

78
00:03:34.805 --> 00:03:37.915
<00:03:34.805>Maybe <00:03:34.980>you're <00:03:35.155>working <00:03:35.330>in <00:03:35.505>a <00:03:35.680>highly <00:03:35.855>regulated <00:03:36.030>environment <00:03:36.205>like

79
00:03:37.915 --> 00:03:39.915
<00:03:37.915>banks, <00:03:38.381>healthcare, <00:03:38.848>public <00:03:39.315>sector.

80
00:03:40.155 --> 00:03:41.915
<00:03:40.155>I <00:03:40.310>think <00:03:40.466>I've <00:03:40.621>talked <00:03:40.777>to <00:03:40.932>many <00:03:41.088>guys <00:03:41.243>who <00:03:41.399>work <00:03:41.555>there

81
00:03:41.915 --> 00:03:44.635
<00:03:41.915>and <00:03:42.090>they're <00:03:42.265>really <00:03:42.440>interested <00:03:42.614>in <00:03:42.790>the <00:03:42.965>second <00:03:43.140>cloud <00:03:43.315>services

82
00:03:44.635 --> 00:03:46.595
<00:03:44.635>because <00:03:44.915>we <00:03:45.195>are <00:03:45.475>selling <00:03:45.755>data <00:03:46.035>sovereignty.

83
00:03:46.855 --> 00:03:48.195
<00:03:46.855>All <00:03:47.055>the <00:03:47.257>data <00:03:47.458>states <00:03:47.659>here <00:03:47.859>in <00:03:48.060>Europe

84
00:03:49.775 --> 00:03:53.155
<00:03:49.775>and <00:03:49.930>we <00:03:50.086>fulfill <00:03:50.241>all <00:03:50.397>the <00:03:50.552>requirements <00:03:50.708>like <00:03:50.863>a <00:03:51.019>C <00:03:51.175>five

85
00:03:53.475 --> 00:03:55.475
<00:03:53.475>certification <00:03:53.825>if <00:03:54.174>you <00:03:54.525>know <00:03:54.875>that

86
00:03:55.615 --> 00:03:57.115
<00:03:55.615>and <00:03:55.885>are <00:03:56.155>working <00:03:56.425>in <00:03:56.695>this <00:03:56.965>environment.

87
00:03:57.455 --> 00:03:58.915
<00:03:57.454>So <00:03:57.717>there's <00:03:57.980>no <00:03:58.243>LA <00:03:58.506>vendor <00:03:58.769>login.

88
00:03:59.215 --> 00:04:01.235
<00:03:59.215>You <00:03:59.390>can <00:03:59.565>host <00:03:59.740>any <00:03:59.915>open <00:04:00.090>source <00:04:00.265>model <00:04:00.440>there <00:04:00.615>is.

89
00:04:01.375 --> 00:04:03.715
<00:04:01.375>So <00:04:01.515>in <00:04:01.655>hugging <00:04:01.795>phase, <00:04:01.935>there's <00:04:02.075>a <00:04:02.215>list <00:04:02.355>of <00:04:02.495>all <00:04:02.635>the, <00:04:02.775>uh,

90
00:04:04.095 --> 00:04:05.155
<00:04:04.095>models <00:04:04.413>you <00:04:04.731>can <00:04:05.049>use.

91
00:04:05.215 --> 00:04:07.155
<00:04:05.215>For <00:04:05.448>example, <00:04:05.681>there's <00:04:05.915>LAMA <00:04:06.148>model <00:04:06.381>from <00:04:06.615>meta,

92
00:04:07.185 --> 00:04:08.475
<00:04:07.185>there's <00:04:07.572>some <00:04:07.959>mytral <00:04:08.346>model.

93
00:04:09.855 --> 00:04:12.355
<00:04:09.855>Um, <00:04:10.030>there's <00:04:10.204>a <00:04:10.380>deep <00:04:10.554>sea <00:04:10.730>which <00:04:10.905>is <00:04:11.079>famous <00:04:11.255>when,

94
00:04:13.015 --> 00:04:15.875
<00:04:13.015>so <00:04:13.131>every <00:04:13.248>time <00:04:13.364>there's <00:04:13.481>some <00:04:13.598>new <00:04:13.714>hype <00:04:13.831>you <00:04:13.948>can <00:04:14.065>just <00:04:14.181>dig <00:04:14.298>it <00:04:14.415>into

95
00:04:15.875 --> 00:04:17.315
<00:04:15.875>your <00:04:16.199>system <00:04:16.523>and <00:04:16.846>use <00:04:17.171>it

96
00:04:17.315 --> 00:04:19.755
<00:04:17.315>because <00:04:17.515>everything <00:04:17.714>is <00:04:17.915>really <00:04:18.115>modular <00:04:18.315>in <00:04:18.515>the <00:04:18.714>architecture

97
00:04:19.895 --> 00:04:20.955
<00:04:19.894>I'm <00:04:20.849>presenting.

98
00:04:22.615 --> 00:04:23.915
<00:04:22.615>Um, <00:04:23.200>and <00:04:23.785>ENCY

99
00:04:23.915 --> 00:04:25.195
<00:04:23.915>and <00:04:24.299>availability <00:04:24.683>control, <00:04:25.067>some

100
00:04:25.195 --> 00:04:27.035
<00:04:25.195>of <00:04:25.350>your <00:04:25.506>applications <00:04:25.661>are <00:04:25.817>running <00:04:25.972>in <00:04:26.128>your <00:04:26.283>own <00:04:26.439>data <00:04:26.594>center

101
00:04:27.455 --> 00:04:29.355
<00:04:27.455>and <00:04:27.630>you <00:04:27.805>want <00:04:27.979>to <00:04:28.155>have <00:04:28.330>the <00:04:28.505>LLM <00:04:28.680>the <00:04:28.854>same

102
00:04:29.355 --> 00:04:31.355
<00:04:29.355>as <00:04:29.555>the <00:04:29.755>database <00:04:29.955>very <00:04:30.155>near <00:04:30.355>to <00:04:30.555>the <00:04:30.755>application.

103
00:04:31.375 --> 00:04:33.035
<00:04:31.375>So <00:04:31.530>if <00:04:31.686>you <00:04:31.841>host <00:04:31.997>it <00:04:32.152>in <00:04:32.308>your <00:04:32.463>own <00:04:32.619>data <00:04:32.775>center

104
00:04:33.035 --> 00:04:35.515
<00:04:33.034>as <00:04:33.190>well <00:04:33.346>your <00:04:33.501>application, <00:04:33.657>then <00:04:33.812>you're <00:04:33.968>saving <00:04:34.123>a <00:04:34.279>little <00:04:34.434>bit

105
00:04:35.515 --> 00:04:36.595
<00:04:35.515>of <00:04:36.001>network <00:04:36.487>latency.

106
00:04:38.855 --> 00:04:41.555
<00:04:38.855>So <00:04:39.010>I <00:04:39.166>don't <00:04:39.321>want <00:04:39.477>to <00:04:39.632>waste <00:04:39.788>a <00:04:39.943>lot <00:04:40.099>of <00:04:40.255>time

107
00:04:42.095 --> 00:04:43.755
<00:04:42.095>of <00:04:42.445>doing <00:04:42.795>this <00:04:43.145>regularity <00:04:43.495>stuff.

108
00:04:44.195 --> 00:04:45.715
<00:04:44.195>I <00:04:44.468>want <00:04:44.742>to <00:04:45.015>rather <00:04:45.289>go <00:04:45.563>technical

109
00:04:46.095 --> 00:04:49.755
<00:04:46.095>and <00:04:46.235>check <00:04:46.375>what <00:04:46.515>we <00:04:46.655>can <00:04:46.795>do <00:04:46.935>to <00:04:47.075>scale <00:04:47.215>this <00:04:47.355>LM <00:04:47.495>horizontally.

110
00:04:50.355 --> 00:04:53.075
<00:04:50.355>'cause <00:04:50.462>at <00:04:50.570>the <00:04:50.678>end <00:04:50.785>of <00:04:50.893>the <00:04:51.001>day <00:04:51.108>an <00:04:51.216>LLM <00:04:51.324>is <00:04:51.431>just <00:04:51.539>the <00:04:51.647>rest <00:04:51.755>API.

111
00:04:53.465 --> 00:04:55.995
<00:04:53.465>Then <00:04:53.640>I'm <00:04:53.815>looking <00:04:53.990>through <00:04:54.165>my <00:04:54.340>software <00:04:54.515>engineering <00:04:54.690>lens, <00:04:54.865>okay,

112
00:04:55.995 --> 00:04:59.075
<00:04:55.995>what <00:04:56.150>can <00:04:56.306>we <00:04:56.461>do <00:04:56.617>to <00:04:56.772>scale <00:04:56.928>this <00:04:57.083>synchronous <00:04:57.239>rest <00:04:57.395>API,

113
00:04:59.365 --> 00:05:02.315
<00:04:59.365>which <00:04:59.520>is <00:04:59.676>very <00:04:59.831>stateless, <00:04:59.987>it's <00:05:00.142>doing <00:05:00.298>a <00:05:00.453>lot <00:05:00.609>of <00:05:00.765>compute.

114
00:05:02.455 --> 00:05:03.715
<00:05:02.455>So <00:05:02.681>I'm <00:05:02.908>looking <00:05:03.135>into <00:05:03.362>my <00:05:03.589>box.

115
00:05:03.785 --> 00:05:06.875
<00:05:03.785>Then <00:05:03.925>I <00:05:04.065>put <00:05:04.205>out <00:05:04.345>a <00:05:04.485>wall <00:05:04.625>and <00:05:04.765>see, <00:05:04.905>ah, <00:05:05.045>casting, <00:05:05.185>obviously

116
00:05:07.655 --> 00:05:09.195
<00:05:07.655>paging, <00:05:09.041>um,

117
00:05:10.345 --> 00:05:12.875
<00:05:10.345>splitting <00:05:10.578>up <00:05:10.811>the <00:05:11.045>work <00:05:11.278>to <00:05:11.511>multiple <00:05:11.745>workers.

118
00:05:13.055 --> 00:05:16.475
<00:05:13.055>So <00:05:13.230>concurrency. <00:05:13.405>So <00:05:13.580>we <00:05:13.755>need <00:05:13.930>to <00:05:14.105>find <00:05:14.280>an <00:05:14.455>architecture

119
00:05:16.475 --> 00:05:18.435
<00:05:16.475>that <00:05:16.755>fulfills <00:05:17.035>all <00:05:17.315>of <00:05:17.595>these <00:05:17.875>requirements

120
00:05:18.815 --> 00:05:21.635
<00:05:18.815>and <00:05:19.015>then <00:05:19.215>we <00:05:19.415>can <00:05:19.615>easily <00:05:19.815>hor <00:05:20.015>horizontal <00:05:20.215>scale.

121
00:05:22.495 --> 00:05:25.395
<00:05:22.495>Um, <00:05:22.635>the <00:05:22.775>LLM, <00:05:22.915>which <00:05:23.055>is <00:05:23.195>basically <00:05:23.335>arrest <00:05:23.475>API, <00:05:23.615>so <00:05:23.755>to <00:05:23.895>say.

122
00:05:25.855 --> 00:05:28.875
<00:05:25.855>So <00:05:25.962>first <00:05:26.070>we <00:05:26.178>need <00:05:26.285>to <00:05:26.393>take <00:05:26.501>care <00:05:26.608>of <00:05:26.716>where <00:05:26.824>do <00:05:26.931>we <00:05:27.039>get <00:05:27.147>our <00:05:27.255>models.

123
00:05:30.285 --> 00:05:34.195
<00:05:30.285>There <00:05:30.460>are <00:05:30.635>multiple <00:05:30.810>options <00:05:30.985>in <00:05:31.160>the <00:05:31.335>open <00:05:31.510>source <00:05:31.685>area

124
00:05:34.535 --> 00:05:35.595
<00:05:34.534>for <00:05:35.489>lms.

125
00:05:35.595 --> 00:05:36.675
<00:05:35.595>There <00:05:35.838>you <00:05:36.081>see <00:05:36.324>hugging <00:05:36.567>face

126
00:05:36.695 --> 00:05:38.635
<00:05:36.695>and <00:05:36.895>under <00:05:37.094>the <00:05:37.295>hood <00:05:37.495>it's <00:05:37.695>basically <00:05:37.895>using <00:05:38.094>object

127
00:05:38.635 --> 00:05:39.835
<00:05:38.635>storage <00:05:39.175>or <00:05:39.715>CDM.

128
00:05:40.215 --> 00:05:43.195
<00:05:40.215>So <00:05:40.415>if <00:05:40.615>you're <00:05:40.815>building <00:05:41.015>your <00:05:41.215>machine <00:05:41.415>learning <00:05:41.615>model,

129
00:05:43.215 --> 00:05:45.235
<00:05:43.215>you <00:05:43.390>probably <00:05:43.565>want <00:05:43.740>to <00:05:43.915>use <00:05:44.090>these <00:05:44.265>options <00:05:44.440>as <00:05:44.615>well.

130
00:05:46.455 --> 00:05:49.315
<00:05:46.455>So <00:05:46.582>in <00:05:46.709>an <00:05:46.836>inference <00:05:46.964>engine, <00:05:47.091>this <00:05:47.218>is <00:05:47.345>doing <00:05:47.473>a <00:05:47.600>lot <00:05:47.727>of <00:05:47.854>tricks,

131
00:05:49.315 --> 00:05:50.315
<00:05:49.315>like <00:05:49.615>I <00:05:49.915>already <00:05:50.215>mentioned,

132
00:05:50.385 --> 00:05:52.515
<00:05:50.385>splitting <00:05:50.618>up <00:05:50.851>the <00:05:51.085>work <00:05:51.318>into <00:05:51.551>multiple <00:05:51.784>frets,

133
00:05:52.565 --> 00:05:55.955
<00:05:52.565>doing <00:05:52.765>the <00:05:52.965>caching, <00:05:53.165>and <00:05:53.365>there <00:05:53.565>are <00:05:53.765>a <00:05:53.965>couple

134
00:05:55.955 --> 00:05:56.955
<00:05:55.955>of <00:05:56.180>other <00:05:56.405>tricks <00:05:56.630>it's <00:05:56.854>doing.

135
00:05:58.505 --> 00:05:59.925
<00:05:58.505>You <00:05:58.718>may <00:05:58.931>want <00:05:59.144>to <00:05:59.357>build <00:05:59.570>it <00:05:59.783>yourself.

136
00:05:59.985 --> 00:06:02.165
<00:05:59.985>You <00:06:00.160>may <00:06:00.335>want <00:06:00.510>to <00:06:00.685>use <00:06:00.860>an <00:06:01.035>open <00:06:01.210>source <00:06:01.385>solution.

137
00:06:04.745 --> 00:06:05.885
<00:06:04.745>So <00:06:04.916>we <00:06:05.087>are <00:06:05.258>seeing <00:06:05.429>what <00:06:05.600>it <00:06:05.771>does.

138
00:06:06.105 --> 00:06:08.325
<00:06:06.105>The <00:06:06.305>next <00:06:06.505>step <00:06:06.705>is <00:06:06.905>we <00:06:07.105>have <00:06:07.305>multiple <00:06:07.505>machines

139
00:06:08.585 --> 00:06:10.885
<00:06:08.585>and <00:06:08.740>the <00:06:08.896>model <00:06:09.051>that <00:06:09.207>is <00:06:09.362>downloaded <00:06:09.518>somewhere, <00:06:09.673>then <00:06:09.829>we <00:06:09.984>need

140
00:06:10.885 --> 00:06:15.045
<00:06:10.885>to <00:06:11.040>store <00:06:11.196>it <00:06:11.351>that <00:06:11.507>to <00:06:11.662>uh, <00:06:11.818>the <00:06:11.973>machine <00:06:12.129>as <00:06:12.284>well.

141
00:06:15.665 --> 00:06:17.885
<00:06:15.665>So <00:06:15.805>the <00:06:15.945>machine <00:06:16.085>need <00:06:16.225>to <00:06:16.365>be <00:06:16.505>able <00:06:16.645>to <00:06:16.785>load <00:06:16.925>these <00:06:17.065>weights.

142
00:06:17.885 --> 00:06:20.805
<00:06:17.885>We <00:06:18.025>are <00:06:18.164>downloading, <00:06:18.305>they <00:06:18.445>have, <00:06:18.585>we <00:06:18.724>have <00:06:18.865>a <00:06:19.005>couple <00:06:19.145>of <00:06:19.284>options

143
00:06:20.825 --> 00:06:22.285
<00:06:20.825>and <00:06:21.044>they <00:06:21.263>all <00:06:21.482>have <00:06:21.701>the <00:06:21.920>trade <00:06:22.139>offs.

144
00:06:24.115 --> 00:06:26.485
<00:06:24.115>Next <00:06:24.255>thing, <00:06:24.395>if <00:06:24.535>you <00:06:24.675>are <00:06:24.815>providing <00:06:24.955>a <00:06:25.095>service <00:06:25.235>to <00:06:25.375>the <00:06:25.515>customer,

145
00:06:26.665 --> 00:06:28.525
<00:06:26.665>we <00:06:26.820>also <00:06:26.976>want <00:06:27.131>to <00:06:27.287>build <00:06:27.442>a <00:06:27.598>customer <00:06:27.753>to <00:06:27.909>use <00:06:28.065>it

146
00:06:28.525 --> 00:06:30.205
<00:06:28.525>because <00:06:28.875>free <00:06:29.224>services <00:06:29.575>are <00:06:29.924>great.

147
00:06:30.905 --> 00:06:32.765
<00:06:30.905>For <00:06:31.080>example, <00:06:31.255>if <00:06:31.429>you're <00:06:31.604>Google, <00:06:31.780>you <00:06:31.955>can <00:06:32.130>afford <00:06:32.304>that.

148
00:06:32.945 --> 00:06:35.605
<00:06:32.945>But <00:06:33.072>as <00:06:33.199>a <00:06:33.326>business <00:06:33.454>you <00:06:33.581>really <00:06:33.708>want <00:06:33.835>to <00:06:33.963>get <00:06:34.090>the <00:06:34.217>revenue <00:06:34.344>back.

149
00:06:35.665 --> 00:06:38.405
<00:06:35.665>You <00:06:35.792>have <00:06:35.919>to <00:06:36.046>pay <00:06:36.174>your <00:06:36.301>employees, <00:06:36.428>you <00:06:36.555>have <00:06:36.683>to <00:06:36.810>pay <00:06:36.937>the, <00:06:37.065>uh,

150
00:06:38.555 --> 00:06:40.165
<00:06:38.555>GPUs <00:06:38.905>which <00:06:39.255>are <00:06:39.605>really <00:06:39.955>costly.

151
00:06:41.025 --> 00:06:43.805
<00:06:41.025>So <00:06:41.141>we <00:06:41.258>want <00:06:41.375>to <00:06:41.491>pay, <00:06:41.608>so <00:06:41.724>we <00:06:41.841>want <00:06:41.958>to <00:06:42.075>build <00:06:42.191>a <00:06:42.308>customer <00:06:42.424>with

152
00:06:44.825 --> 00:06:46.845
<00:06:44.825>the <00:06:45.058>uh, <00:06:45.291>provider <00:06:45.525>functionality <00:06:45.758>of <00:06:45.991>the <00:06:46.224>model.

153
00:06:48.275 --> 00:06:51.845
<00:06:48.275>Next, <00:06:48.474>how <00:06:48.674>do <00:06:48.875>we <00:06:49.075>handle, <00:06:49.275>uh, <00:06:49.474>more <00:06:49.674>load

154
00:06:51.945 --> 00:06:55.005
<00:06:51.945>and <00:06:52.145>provide <00:06:52.344>prevent <00:06:52.545>overloading <00:06:52.745>from <00:06:52.945>a <00:06:53.145>single <00:06:53.344>user?

155
00:06:55.515 --> 00:06:57.885
<00:06:55.515>This <00:06:55.670>goes <00:06:55.826>into <00:06:55.981>the <00:06:56.137>direction <00:06:56.292>of <00:06:56.448>freight <00:06:56.603>limiting <00:06:56.759>of <00:06:56.914>course.

156
00:06:58.305 --> 00:07:01.045
<00:06:58.305>So <00:06:58.505>if <00:06:58.705>you <00:06:58.905>have <00:06:59.105>any <00:06:59.305>shared <00:06:59.505>SaaS <00:06:59.705>service,

157
00:07:01.705 --> 00:07:03.125
<00:07:01.705>you <00:07:01.918>want <00:07:02.131>to <00:07:02.344>look <00:07:02.557>into <00:07:02.770>this <00:07:02.983>topic.

158
00:07:03.575 --> 00:07:07.645
<00:07:03.575>There <00:07:03.750>are <00:07:03.925>different <00:07:04.099>algorithms <00:07:04.275>And <00:07:04.450>we <00:07:04.625>will <00:07:04.800>look <00:07:04.974>into

159
00:07:07.645 --> 00:07:10.645
<00:07:07.645>that <00:07:07.844>in <00:07:08.044>a <00:07:08.245>small, <00:07:08.445>in <00:07:08.645>small <00:07:08.844>time <00:07:09.044>amount

160
00:07:11.155 --> 00:07:12.845
<00:07:11.155>time <00:07:11.434>in <00:07:11.715>small <00:07:11.994>amount <00:07:12.275>of <00:07:12.554>time.

161
00:07:14.425 --> 00:07:17.405
<00:07:14.425>So <00:07:14.565>next <00:07:14.705>we <00:07:14.845>need <00:07:14.985>to <00:07:15.125>take <00:07:15.265>care <00:07:15.405>about <00:07:15.545>the <00:07:15.685>security <00:07:15.825>thing.

162
00:07:17.545 --> 00:07:18.925
<00:07:17.545>How <00:07:17.793>do <00:07:18.041>we <00:07:18.290>authenticate <00:07:18.538>a <00:07:18.787>user?

163
00:07:18.935 --> 00:07:21.445
<00:07:18.935>Maybe <00:07:19.215>there's <00:07:19.495>also <00:07:19.775>some <00:07:20.055>offer <00:07:20.335>authorization,

164
00:07:21.655 --> 00:07:23.845
<00:07:21.655>maybe <00:07:21.888>some <00:07:22.121>users <00:07:22.354>want <00:07:22.588>to <00:07:22.821>get <00:07:23.054>access

165
00:07:23.845 --> 00:07:25.325
<00:07:23.845>to <00:07:24.111>certain <00:07:24.377>routes, <00:07:24.644>to <00:07:24.910>certain <00:07:25.177>models.

166
00:07:27.545 --> 00:07:30.165
<00:07:27.545>And <00:07:27.720>then <00:07:27.895>we <00:07:28.070>check <00:07:28.245>what <00:07:28.420>options <00:07:28.595>do <00:07:28.770>we <00:07:28.945>have

167
00:07:31.875 --> 00:07:34.885
<00:07:31.875>Next <00:07:32.030>we <00:07:32.186>want <00:07:32.341>to <00:07:32.497>distribute <00:07:32.652>the <00:07:32.808>load <00:07:32.963>over <00:07:33.119>multiple <00:07:33.275>machines.

168
00:07:35.395 --> 00:07:37.885
<00:07:35.395>Then <00:07:35.570>we <00:07:35.745>need <00:07:35.919>to <00:07:36.094>think <00:07:36.270>about <00:07:36.445>some <00:07:36.620>cannery <00:07:36.794>deployments.

169
00:07:39.065 --> 00:07:42.565
<00:07:39.065>Um, <00:07:39.220>what <00:07:39.376>kind <00:07:39.531>of <00:07:39.687>outer <00:07:39.842>scaling <00:07:39.998>mechanisms <00:07:40.153>do <00:07:40.309>we <00:07:40.465>have?

170
00:07:44.105 --> 00:07:47.325
<00:07:44.105>For <00:07:44.260>example, <00:07:44.416>the <00:07:44.571>easiest <00:07:44.727>solution <00:07:44.882>is <00:07:45.038>based <00:07:45.193>on <00:07:45.349>CPU <00:07:45.505>load

171
00:07:47.465 --> 00:07:52.285
<00:07:47.465>or <00:07:47.581>memory <00:07:47.698>load <00:07:47.815>or <00:07:47.931>a <00:07:48.048>GU <00:07:48.165>memory <00:07:48.281>load, <00:07:48.398>but <00:07:48.515>there <00:07:48.631>are <00:07:48.748>many <00:07:48.865>more.

172
00:07:52.505 --> 00:07:53.765
<00:07:52.505>And <00:07:52.731>this <00:07:52.958>goes <00:07:53.185>into <00:07:53.412>the <00:07:53.639>direction

173
00:07:53.765 --> 00:07:55.565
<00:07:53.765>of <00:07:54.231>distributed <00:07:54.698>inference <00:07:55.164>engines.

174
00:07:56.265 --> 00:07:58.645
<00:07:56.265>And <00:07:56.420>finally, <00:07:56.576>we <00:07:56.731>don't <00:07:56.887>only <00:07:57.042>want <00:07:57.198>to <00:07:57.353>host <00:07:57.509>one <00:07:57.664>model,

175
00:07:58.665 --> 00:08:01.485
<00:07:58.665>but <00:07:58.865>many <00:07:59.065>models <00:07:59.265>and <00:07:59.465>the <00:07:59.665>model <00:07:59.865>is <00:08:00.065>selected

176
00:08:01.585 --> 00:08:02.645
<00:08:01.585>in <00:08:01.902>the <00:08:02.221>request <00:08:02.539>body.

177
00:08:02.825 --> 00:08:05.045
<00:08:02.825>So <00:08:03.000>we <00:08:03.175>need <00:08:03.349>to <00:08:03.525>have <00:08:03.700>some <00:08:03.875>kind <00:08:04.050>of <00:08:04.224>component

178
00:08:05.235 --> 00:08:06.525
<00:08:05.235>that <00:08:05.622>routes <00:08:06.009>the <00:08:06.395>request

179
00:08:06.745 --> 00:08:09.805
<00:08:06.745>to <00:08:06.900>the <00:08:07.056>right <00:08:07.211>virtual <00:08:07.367>machine <00:08:07.522>where <00:08:07.678>the <00:08:07.833>model <00:08:07.989>is <00:08:08.145>running.

180
00:08:11.345 --> 00:08:14.125
<00:08:11.345>And <00:08:11.500>this <00:08:11.656>is <00:08:11.811>the <00:08:11.967>component <00:08:12.122>I <00:08:12.278>have <00:08:12.433>called <00:08:12.589>model <00:08:12.745>router.

181
00:08:16.975 --> 00:08:20.395
<00:08:16.975>So <00:08:17.130>here <00:08:17.286>the <00:08:17.441>overview. <00:08:17.597>Let's <00:08:17.752>think <00:08:17.908>about <00:08:18.063>a <00:08:18.219>model <00:08:18.375>provider.

182
00:08:20.425 --> 00:08:21.435
<00:08:20.425>It's <00:08:20.728>very, <00:08:21.031>very <00:08:21.334>simple.

183
00:08:21.975 --> 00:08:25.435
<00:08:21.975>So <00:08:22.130>an <00:08:22.286>LLM <00:08:22.441>is <00:08:22.597>hugely <00:08:22.752>quite <00:08:22.908>large <00:08:23.063>on <00:08:23.219>the <00:08:23.375>weights.

184
00:08:25.795 --> 00:08:28.315
<00:08:25.795>A <00:08:25.995>small <00:08:26.195>model <00:08:26.395>is <00:08:26.595>already <00:08:26.795>at <00:08:26.995>20 <00:08:27.195>gigabytes.

185
00:08:28.905 --> 00:08:31.715
<00:08:28.905>Some <00:08:29.060>larger <00:08:29.216>models <00:08:29.371>like <00:08:29.527>a <00:08:29.682>llama <00:08:29.838>70 <00:08:29.993>B <00:08:30.149>for <00:08:30.304>example,

186
00:08:31.775 --> 00:08:33.195
<00:08:31.775>has <00:08:32.414>200 <00:08:33.053>gigabytes

187
00:08:33.195 --> 00:08:35.915
<00:08:33.195>and <00:08:33.335>we <00:08:33.475>need <00:08:33.615>to <00:08:33.755>somehow <00:08:33.895>download <00:08:34.035>it <00:08:34.175>to <00:08:34.315>our <00:08:34.455>local <00:08:34.595>storage.

188
00:08:36.255 --> 00:08:39.955
<00:08:36.255>So <00:08:36.371>DVM <00:08:36.488>can <00:08:36.605>use <00:08:36.721>it <00:08:36.838>and <00:08:36.955>start <00:08:37.071>a <00:08:37.188>model <00:08:37.304>based <00:08:37.421>on <00:08:37.538>the <00:08:37.655>spas.

189
00:08:41.275 --> 00:08:43.655
<00:08:41.275>As <00:08:41.449>already <00:08:41.625>mentioned, <00:08:41.799>most <00:08:41.975>models <00:08:42.150>are <00:08:42.324>stored <00:08:42.500>on <00:08:42.674>hugging

190
00:08:43.665 --> 00:08:45.335
<00:08:43.664>phase <00:08:43.944>with <00:08:44.224>some <00:08:44.505>kind <00:08:44.785>of <00:08:45.064>discovery

191
00:08:45.635 --> 00:08:46.975
<00:08:45.635>for <00:08:45.936>open <00:08:46.238>source <00:08:46.539>language <00:08:46.841>models.

192
00:08:47.135 --> 00:08:49.895
<00:08:47.135>I <00:08:47.290>think <00:08:47.446>they <00:08:47.601>host <00:08:47.757>even <00:08:47.912>more <00:08:48.068>than <00:08:48.223>only <00:08:48.379>large <00:08:48.535>language

193
00:08:49.895 --> 00:08:51.255
<00:08:49.895>models <00:08:50.303>under <00:08:50.711>the <00:08:51.119>hood.

194
00:08:51.255 --> 00:08:52.975
<00:08:51.255>They're <00:08:51.535>using <00:08:51.814>of <00:08:52.095>course, <00:08:52.375>object <00:08:52.655>storage.

195
00:08:53.455 --> 00:08:55.565
<00:08:53.455>I <00:08:53.655>think <00:08:53.855>this <00:08:54.055>is <00:08:54.255>a <00:08:54.455>pretty <00:08:54.655>famous <00:08:54.855>solution

196
00:08:55.665 --> 00:08:58.085
<00:08:55.665>to <00:08:55.944>store <00:08:56.224>media <00:08:56.505>files <00:08:56.785>in <00:08:57.064>general

197
00:08:59.585 --> 00:09:03.165
<00:08:59.585>and <00:08:59.740>to <00:08:59.896>optimize <00:09:00.051>the <00:09:00.207>download <00:09:00.362>we <00:09:00.518>can <00:09:00.673>use <00:09:00.829>A <00:09:00.985>CDN.

198
00:09:03.315 --> 00:09:05.525
<00:09:03.315>It's <00:09:03.515>a <00:09:03.715>cache <00:09:03.915>for <00:09:04.115>the <00:09:04.315>real <00:09:04.515>object <00:09:04.715>storage,

199
00:09:06.785 --> 00:09:08.205
<00:09:06.785>but <00:09:07.040>this <00:09:07.296>all <00:09:07.551>has <00:09:07.807>trade <00:09:08.063>offs.

200
00:09:08.205 --> 00:09:10.445
<00:09:08.205>Maybe <00:09:08.380>you <00:09:08.555>have <00:09:08.730>your <00:09:08.905>own <00:09:09.080>object <00:09:09.255>storage <00:09:09.430>host <00:09:09.605>somewhere

201
00:09:10.445 --> 00:09:11.565
<00:09:10.445>because <00:09:10.646>you <00:09:10.848>train <00:09:11.049>your <00:09:11.251>own <00:09:11.453>models.

202
00:09:11.795 --> 00:09:14.165
<00:09:11.795>Then <00:09:11.969>you <00:09:12.145>think <00:09:12.319>about, <00:09:12.495>okay, <00:09:12.670>what <00:09:12.844>does <00:09:13.020>it <00:09:13.194>cost

203
00:09:14.185 --> 00:09:15.485
<00:09:14.185>to <00:09:14.574>append <00:09:14.965>A <00:09:15.355>CDN?

204
00:09:15.485 --> 00:09:16.725
<00:09:15.485>Do <00:09:15.708>I <00:09:15.931>even <00:09:16.154>need <00:09:16.377>A <00:09:16.601>CDN?

205
00:09:16.725 --> 00:09:21.325
<00:09:16.725>Do <00:09:16.865>I <00:09:17.005>need <00:09:17.145>to <00:09:17.285>um, <00:09:17.425>prove <00:09:17.565>the <00:09:17.705>download <00:09:17.845>latency <00:09:17.985>with <00:09:18.125>it?

206
00:09:22.385 --> 00:09:25.325
<00:09:22.385>So <00:09:22.585>this <00:09:22.785>is <00:09:22.985>the <00:09:23.185>typical <00:09:23.385>arguments <00:09:23.585>we <00:09:23.785>need

207
00:09:25.325 --> 00:09:27.685
<00:09:25.325>to <00:09:25.525>think <00:09:25.725>about <00:09:25.925>while <00:09:26.125>introducing <00:09:26.325>any <00:09:26.525>kind <00:09:26.725>of

208
00:09:29.975 --> 00:09:31.365
<00:09:29.975>Technology <00:09:30.392>to <00:09:30.809>our <00:09:31.226>stack.

209
00:09:31.825 --> 00:09:33.925
<00:09:31.825>For <00:09:32.000>example, <00:09:32.175>what <00:09:32.350>is <00:09:32.525>the <00:09:32.700>rate <00:09:32.875>limitings <00:09:33.050>of <00:09:33.225>S3?

210
00:09:34.085 --> 00:09:38.165
<00:09:34.085>I <00:09:34.240>think <00:09:34.396>you <00:09:34.551>have <00:09:34.707>around <00:09:34.862>3,500 <00:09:35.018>reads <00:09:35.173>for <00:09:35.329>your <00:09:35.485>bucket

211
00:09:38.385 --> 00:09:40.085
<00:09:38.385>and <00:09:39.085>5,000 <00:09:39.785>rides.

212
00:09:40.735 --> 00:09:44.165
<00:09:40.735>Maybe <00:09:40.968>it's <00:09:41.201>5,000 <00:09:41.435>reads <00:09:41.668>and <00:09:41.901>3,500 <00:09:42.135>rides.

213
00:09:44.625 --> 00:09:47.285
<00:09:44.625>Do <00:09:44.752>you <00:09:44.879>go <00:09:45.006>over <00:09:45.134>this <00:09:45.261>limit? <00:09:45.388>What <00:09:45.515>can <00:09:45.643>you <00:09:45.770>do <00:09:45.897>about <00:09:46.025>it?

214
00:09:47.285 --> 00:09:50.645
<00:09:47.285>Then <00:09:47.459>you <00:09:47.635>introduce <00:09:47.810>the <00:09:47.985>cash <00:09:48.160>with <00:09:48.334>the <00:09:48.510>CDN <00:09:48.685>basically,

215
00:09:52.215 --> 00:09:54.765
<00:09:52.215>since <00:09:52.390>we <00:09:52.565>are <00:09:52.740>really <00:09:52.915>critical <00:09:53.090>on <00:09:53.265>the <00:09:53.440>download <00:09:53.615>path

216
00:09:54.765 --> 00:09:58.685
<00:09:54.765>and <00:09:54.881>we <00:09:54.998>want <00:09:55.115>to <00:09:55.231>start <00:09:55.348>a <00:09:55.465>new <00:09:55.581>machine <00:09:55.698>with <00:09:55.814>our <00:09:55.931>model, <00:09:56.048>we <00:09:56.165>want

217
00:09:58.685 --> 00:10:00.165
<00:09:58.685>to <00:09:58.906>think <00:09:59.128>about, <00:09:59.351>okay, <00:09:59.573>what's <00:09:59.795>the <00:10:00.016>SLA?

218
00:10:00.385 --> 00:10:02.365
<00:10:00.385>We <00:10:00.665>basically <00:10:00.944>want <00:10:01.225>to <00:10:01.505>have <00:10:01.785>100%,

219
00:10:02.385 --> 00:10:03.725
<00:10:02.385>but <00:10:02.626>it's <00:10:02.867>almost <00:10:03.108>never <00:10:03.349>the <00:10:03.591>case.

220
00:10:04.225 --> 00:10:06.405
<00:10:04.225>So <00:10:04.425>maybe <00:10:04.625>want <00:10:04.825>to <00:10:05.025>have <00:10:05.225>the <00:10:05.425>model <00:10:05.625>local

221
00:10:07.465 --> 00:10:08.925
<00:10:07.465>and <00:10:07.652>this <00:10:07.840>is <00:10:08.028>one <00:10:08.215>of <00:10:08.403>the <00:10:08.591>next <00:10:08.779>parts.

222
00:10:11.905 --> 00:10:15.445
<00:10:11.905>So <00:10:12.105>the <00:10:12.305>inference <00:10:12.505>engine, <00:10:12.704>there <00:10:12.905>are <00:10:13.105>a <00:10:13.305>couple

223
00:10:15.445 --> 00:10:16.685
<00:10:15.445>of <00:10:15.817>open <00:10:16.189>source <00:10:16.560>providers.

224
00:10:17.075 --> 00:10:19.325
<00:10:17.075>What <00:10:17.250>the <00:10:17.425>inference <00:10:17.600>engine <00:10:17.775>is <00:10:17.950>doing <00:10:18.125>is <00:10:18.300>a <00:10:18.475>lot

225
00:10:19.325 --> 00:10:20.925
<00:10:19.325>of <00:10:19.675>optimization <00:10:20.025>around <00:10:20.375>the <00:10:20.725>model.

226
00:10:21.185 --> 00:10:23.525
<00:10:21.185>As <00:10:21.359>I <00:10:21.535>already <00:10:21.709>mentioned. <00:10:21.885>It <00:10:22.060>has <00:10:22.234>a <00:10:22.410>cache <00:10:22.584>mechanism.

227
00:10:24.175 --> 00:10:26.405
<00:10:24.175>Since <00:10:24.349>the <00:10:24.525>LLM <00:10:24.699>is <00:10:24.875>doing <00:10:25.050>a <00:10:25.224>lot <00:10:25.400>of <00:10:25.574>work

228
00:10:27.225 --> 00:10:29.285
<00:10:27.225>and <00:10:27.575>to <00:10:27.925>optimize <00:10:28.275>the <00:10:28.625>work

229
00:10:29.285 --> 00:10:31.445
<00:10:29.285>that <00:10:29.485>we <00:10:29.685>are <00:10:29.885>not <00:10:30.084>doing <00:10:30.285>it <00:10:30.485>duplicate <00:10:30.685>times,

230
00:10:32.945 --> 00:10:35.645
<00:10:32.945>we <00:10:33.072>can <00:10:33.199>save <00:10:33.326>this <00:10:33.454>compute <00:10:33.581>heavy <00:10:33.708>task <00:10:33.835>in <00:10:33.963>some <00:10:34.090>kind <00:10:34.217>of <00:10:34.345>memory

231
00:10:35.985 --> 00:10:38.725
<00:10:35.985>to <00:10:36.140>have <00:10:36.296>it <00:10:36.451>there <00:10:36.607>and <00:10:36.762>every <00:10:36.918>time <00:10:37.073>it <00:10:37.229>appears <00:10:37.385>again,

232
00:10:40.095 --> 00:10:41.995
<00:10:40.095>we <00:10:40.250>can <00:10:40.406>read <00:10:40.561>out <00:10:40.717>through <00:10:40.872>the <00:10:41.028>memory <00:10:41.183>and <00:10:41.339>save <00:10:41.495>time.

233
00:10:42.055 --> 00:10:43.755
<00:10:42.055>So <00:10:42.288>one <00:10:42.521>thing <00:10:42.755>is <00:10:42.988>doing <00:10:43.221>is <00:10:43.454>caching,

234
00:10:43.985 --> 00:10:46.155
<00:10:43.985>then <00:10:44.160>it <00:10:44.335>has <00:10:44.510>an <00:10:44.685>internal <00:10:44.860>work <00:10:45.035>of <00:10:45.210>threat <00:10:45.385>tool.

235
00:10:46.225 --> 00:10:48.355
<00:10:46.225>When <00:10:46.425>an <00:10:46.625>request <00:10:46.825>is <00:10:47.025>coming <00:10:47.225>in <00:10:47.425>through <00:10:47.625>this

236
00:10:49.275 --> 00:10:51.755
<00:10:49.275>synchronous <00:10:49.475>rest <00:10:49.675>API, <00:10:49.875>it <00:10:50.074>breaks <00:10:50.275>down <00:10:50.475>the <00:10:50.675>brand

237
00:10:52.175 --> 00:10:54.715
<00:10:52.175>and <00:10:52.375>assign <00:10:52.574>it <00:10:52.775>to <00:10:52.974>some <00:10:53.175>work <00:10:53.375>of <00:10:53.574>threats.

238
00:10:54.815 --> 00:10:56.515
<00:10:54.815>So <00:10:55.281>it's <00:10:55.748>completely <00:10:56.215>asynchronously,

239
00:10:57.215 --> 00:10:59.715
<00:10:57.215>but <00:10:57.495>it <00:10:57.775>also <00:10:58.055>has <00:10:58.335>some <00:10:58.615>downsides.

240
00:11:01.135 --> 00:11:05.275
<00:11:01.135>So <00:11:01.310>if <00:11:01.485>a <00:11:01.660>client <00:11:01.835>is <00:11:02.010>cancel <00:11:02.185>canceling <00:11:02.360>his <00:11:02.535>request,

241
00:11:06.175 --> 00:11:08.675
<00:11:06.175>the <00:11:06.330>work <00:11:06.486>is <00:11:06.641>still <00:11:06.797>in <00:11:06.952>the <00:11:07.108>queue <00:11:07.263>and <00:11:07.419>it's <00:11:07.574>processed

242
00:11:09.415 --> 00:11:12.315
<00:11:09.415>and <00:11:09.531>if <00:11:09.648>the <00:11:09.765>work <00:11:09.881>is <00:11:09.998>done, <00:11:10.115>it's <00:11:10.231>go <00:11:10.348>back <00:11:10.464>to <00:11:10.581>the <00:11:10.698>UH, <00:11:10.814>coordinator

243
00:11:12.335 --> 00:11:14.555
<00:11:12.335>to <00:11:12.535>the <00:11:12.735>scheduler <00:11:12.935>and <00:11:13.135>then <00:11:13.335>it <00:11:13.535>sees, <00:11:13.735>oh,

244
00:11:14.655 --> 00:11:17.195
<00:11:14.655>the <00:11:14.782>rigor <00:11:14.909>is <00:11:15.036>already <00:11:15.164>canceled, <00:11:15.291>so <00:11:15.418>what <00:11:15.545>can <00:11:15.673>I <00:11:15.800>do <00:11:15.927>with <00:11:16.055>it?

245
00:11:18.465 --> 00:11:20.635
<00:11:18.465>This <00:11:18.665>is <00:11:18.865>some <00:11:19.065>uh, <00:11:19.265>problem <00:11:19.465>we <00:11:19.665>encountered <00:11:19.865>while

246
00:11:20.735 --> 00:11:21.795
<00:11:20.735>uh, <00:11:21.053>doing <00:11:21.371>load <00:11:21.689>tests.

247
00:11:24.015 --> 00:11:26.995
<00:11:24.015>Um, <00:11:24.155>the <00:11:24.295>work <00:11:24.435>stays <00:11:24.574>in <00:11:24.715>this <00:11:24.855>queue <00:11:24.995>even <00:11:25.135>though <00:11:25.275>the <00:11:25.415>client

248
00:11:26.995 --> 00:11:29.275
<00:11:26.995>already <00:11:27.228>canceled <00:11:27.461>and <00:11:27.695>cannot <00:11:27.928>use <00:11:28.161>it <00:11:28.395>anymore.

249
00:11:29.665 --> 00:11:30.835
<00:11:29.665>This <00:11:29.928>lead <00:11:30.191>to <00:11:30.454>some, <00:11:30.718>uh,

250
00:11:30.845 --> 00:11:34.275
<00:11:30.845>surprising <00:11:31.020>problems <00:11:31.195>which <00:11:31.370>we <00:11:31.545>can <00:11:31.720>tackle <00:11:31.895>with <00:11:32.070>rate <00:11:32.245>limiting.

251
00:11:35.905 --> 00:11:38.555
<00:11:35.905>This <00:11:36.060>is <00:11:36.216>basically <00:11:36.371>what <00:11:36.527>this <00:11:36.682>inference <00:11:36.838>engines <00:11:36.993>do <00:11:37.149>to <00:11:37.305>optimize

252
00:11:39.055 --> 00:11:41.435
<00:11:39.055>how <00:11:39.229>the <00:11:39.405>model <00:11:39.579>is <00:11:39.755>used <00:11:39.930>in <00:11:40.104>this <00:11:40.280>LLM <00:11:40.454>context.

253
00:11:43.065 --> 00:11:44.275
<00:11:43.065>When <00:11:43.337>we <00:11:43.609>built <00:11:43.881>our <00:11:44.154>service,

254
00:11:44.455 --> 00:11:46.315
<00:11:44.455>we <00:11:44.735>had <00:11:45.015>in <00:11:45.295>mind <00:11:45.575>obviously <00:11:45.855>performance.

255
00:11:46.315 --> 00:11:48.275
<00:11:46.315>Then <00:11:46.595>we <00:11:46.875>looked <00:11:47.155>at <00:11:47.435>multiple <00:11:47.715>benchmarks

256
00:11:49.375 --> 00:11:52.835
<00:11:49.375>and <00:11:49.575>unfortunately <00:11:49.775>they <00:11:49.975>are <00:11:50.175>quite <00:11:50.375>contradicting <00:11:50.575>depending <00:11:50.775>on

257
00:11:52.835 --> 00:11:54.085
<00:11:52.835>who <00:11:53.116>is <00:11:53.397>doing <00:11:53.678>the <00:11:53.960>benchmark.

258
00:11:54.195 --> 00:11:55.565
<00:11:54.195>They <00:11:54.441>have <00:11:54.688>some <00:11:54.934>kind <00:11:55.181>of <00:11:55.428>favorite

259
00:11:56.385 --> 00:12:00.965
<00:11:56.385>and <00:11:56.665>so <00:11:56.944>you <00:11:57.225>can <00:11:57.505>get <00:11:57.785>inspired

260
00:12:01.065 --> 00:12:03.205
<00:12:01.065>by <00:12:01.265>these <00:12:01.465>benchmarks, <00:12:01.665>but <00:12:01.865>always <00:12:02.065>run <00:12:02.265>it <00:12:02.465>yourself

261
00:12:03.795 --> 00:12:06.985
<00:12:03.795>with <00:12:04.145>your <00:12:04.495>own <00:12:04.844>uh, <00:12:05.194>Benchmark.

262
00:12:06.985 --> 00:12:11.945
<00:12:06.985>You <00:12:07.125>graded <00:12:07.265>to <00:12:07.405>have <00:12:07.545>some <00:12:07.685>kind <00:12:07.825>of <00:12:07.965>no <00:12:08.105>biases <00:12:08.245>in <00:12:08.385>there.

263
00:12:12.895 --> 00:12:15.625
<00:12:12.895>Then <00:12:13.069>hardware <00:12:13.245>support <00:12:13.420>we <00:12:13.595>are <00:12:13.770>using <00:12:13.944>and <00:12:14.120>media <00:12:14.295>GPUs.

264
00:12:15.845 --> 00:12:17.585
<00:12:15.845>So <00:12:16.045>almost <00:12:16.245>all <00:12:16.445>of <00:12:16.645>them <00:12:16.845>are <00:12:17.045>supporting <00:12:17.245>it,

265
00:12:18.205 --> 00:12:21.265
<00:12:18.205>but <00:12:18.345>maybe <00:12:18.485>you <00:12:18.625>have <00:12:18.765>some <00:12:18.905>a <00:12:19.045>MD <00:12:19.185>chip <00:12:19.325>for <00:12:19.465>whatever <00:12:19.605>reason.

266
00:12:21.335 --> 00:12:23.505
<00:12:21.335>Then <00:12:21.535>you <00:12:21.735>need <00:12:21.935>to <00:12:22.135>also <00:12:22.335>evaluate <00:12:22.535>that <00:12:22.735>While

267
00:12:24.255 --> 00:12:27.105
<00:12:24.255>comp <00:12:24.455>comparability, <00:12:24.655>we <00:12:24.855>want <00:12:25.055>to <00:12:25.255>host <00:12:25.455>the <00:12:25.655>major

268
00:12:27.615 --> 00:12:28.745
<00:12:27.615>open <00:12:27.869>source <00:12:28.123>model <00:12:28.377>of <00:12:28.632>course,

269
00:12:29.205 --> 00:12:30.905
<00:12:29.205>but <00:12:29.405>we <00:12:29.605>are <00:12:29.805>working <00:12:30.005>in <00:12:30.205>this <00:12:30.405>German <00:12:30.605>market

270
00:12:31.035 --> 00:12:33.945
<00:12:31.035>where <00:12:31.209>we <00:12:31.385>have <00:12:31.560>specialized <00:12:31.735>models <00:12:31.910>for <00:12:32.084>the <00:12:32.260>German <00:12:32.435>language

271
00:12:34.365 --> 00:12:36.785
<00:12:34.365>and <00:12:34.520>we <00:12:34.676>also <00:12:34.831>only <00:12:34.987>want <00:12:35.142>to <00:12:35.298>able <00:12:35.453>to <00:12:35.609>host <00:12:35.765>them

272
00:12:36.785 --> 00:12:40.305
<00:12:36.785>because <00:12:36.959>the <00:12:37.135>customers <00:12:37.310>are <00:12:37.485>required <00:12:37.660>that <00:12:37.834>then <00:12:38.010>we <00:12:38.185>want

273
00:12:40.305 --> 00:12:41.625
<00:12:40.305>to <00:12:40.542>check <00:12:40.780>for <00:12:41.017>the <00:12:41.255>community <00:12:41.492>support.

274
00:12:42.045 --> 00:12:44.305
<00:12:42.045>The <00:12:42.278>uh, <00:12:42.511>inference <00:12:42.745>engine <00:12:42.978>should <00:12:43.211>also <00:12:43.444>be

275
00:12:44.305 --> 00:12:45.505
<00:12:44.305>developed <00:12:44.665>again <00:12:45.025>and <00:12:45.385>again.

276
00:12:45.635 --> 00:12:47.465
<00:12:45.635>There <00:12:45.985>should <00:12:46.335>be <00:12:46.685>security <00:12:47.035>patches

277
00:12:47.965 --> 00:12:49.945
<00:12:47.965>and <00:12:48.315>fixes, <00:12:48.665>updates, <00:12:49.015>new <00:12:49.365>features.

278
00:12:50.525 --> 00:12:52.785
<00:12:50.525>So <00:12:50.758>we <00:12:50.991>rely <00:12:51.225>heavily <00:12:51.458>on <00:12:51.691>the <00:12:51.925>community

279
00:12:53.445 --> 00:12:55.905
<00:12:53.445>and <00:12:53.600>also <00:12:53.756>want <00:12:53.911>to <00:12:54.067>give <00:12:54.222>back <00:12:54.378>something <00:12:54.533>to <00:12:54.689>the <00:12:54.845>community.

280
00:12:55.925 --> 00:12:58.585
<00:12:55.925>If <00:12:56.052>we <00:12:56.179>want <00:12:56.306>to <00:12:56.434>improve <00:12:56.561>something, <00:12:56.688>we <00:12:56.815>should <00:12:56.943>be <00:12:57.070>able <00:12:57.197>to <00:12:57.324>go

281
00:12:58.585 --> 00:13:00.745
<00:12:58.585>to <00:12:58.760>the <00:12:58.935>code <00:12:59.110>base <00:12:59.285>and <00:12:59.460>create <00:12:59.635>a <00:12:59.810>pull <00:12:59.985>request.

282
00:13:02.235 --> 00:13:05.045
<00:13:02.235>Then <00:13:02.390>since <00:13:02.546>we <00:13:02.701>are <00:13:02.857>hosting <00:13:03.012>quite <00:13:03.168>exotic <00:13:03.323>models, <00:13:03.479>we <00:13:03.635>want

283
00:13:05.045 --> 00:13:06.485
<00:13:05.045>to <00:13:05.368>be <00:13:05.693>able <00:13:06.017>to <00:13:06.341>uh,

284
00:13:08.285 --> 00:13:10.045
<00:13:08.285>injected <00:13:08.635>into <00:13:08.985>dis <00:13:09.334>inference <00:13:09.685>engine.

285
00:13:10.065 --> 00:13:11.685
<00:13:10.065>And <00:13:10.298>this <00:13:10.531>is <00:13:10.765>usually <00:13:10.998>done <00:13:11.231>via <00:13:11.465>some

286
00:13:11.685 --> 00:13:13.245
<00:13:11.685>transformation <00:13:12.385>uh, <00:13:13.084>function.

287
00:13:14.545 --> 00:13:16.285
<00:13:14.545>And <00:13:14.685>of <00:13:14.824>course <00:13:14.964>at <00:13:15.104>the <00:13:15.245>end <00:13:15.385>of <00:13:15.525>the <00:13:15.665>day <00:13:15.805>we <00:13:15.944>want

288
00:13:16.285 --> 00:13:17.685
<00:13:16.285>to <00:13:16.495>have <00:13:16.704>an <00:13:16.915>open <00:13:17.125>AI <00:13:17.334>compatible <00:13:17.545>server.

289
00:13:17.785 --> 00:13:19.605
<00:13:17.785>The <00:13:17.985>rest <00:13:18.185>API <00:13:18.385>server, <00:13:18.584>it <00:13:18.785>has <00:13:18.985>a <00:13:19.185>couple

290
00:13:19.625 --> 00:13:22.685
<00:13:19.625>of <00:13:19.800>standard <00:13:19.975>rest <00:13:20.150>API <00:13:20.325>routes <00:13:20.500>like <00:13:20.675>V <00:13:20.850>one <00:13:21.025>completion.

291
00:13:23.065 --> 00:13:25.365
<00:13:23.065>We <00:13:23.298>want <00:13:23.531>chat <00:13:23.765>completion <00:13:23.998>and <00:13:24.231>so <00:13:24.465>on

292
00:13:26.075 --> 00:13:28.445
<00:13:26.075>because <00:13:26.250>we <00:13:26.425>don't <00:13:26.600>really <00:13:26.775>want <00:13:26.950>to <00:13:27.125>implement <00:13:27.300>it <00:13:27.475>ourself.

293
00:13:29.465 --> 00:13:32.285
<00:13:29.465>And <00:13:29.640>the <00:13:29.815>final <00:13:29.990>winner <00:13:30.165>in <00:13:30.340>our <00:13:30.515>evaluation <00:13:30.690>was <00:13:30.865>VLM.

294
00:13:32.355 --> 00:13:35.605
<00:13:32.355>It's <00:13:32.555>quite <00:13:32.755>solid. <00:13:32.955>It <00:13:33.155>never <00:13:33.355>caused <00:13:33.555>any <00:13:33.755>problems.

295
00:13:37.775 --> 00:13:42.715
<00:13:37.775>So <00:13:37.891>now <00:13:38.008>we <00:13:38.125>have <00:13:38.241>downloaded <00:13:38.358>our <00:13:38.475>model <00:13:38.591>to <00:13:38.708>uh, <00:13:38.824>the <00:13:38.941>uh, <00:13:39.058>VM <00:13:39.175>so

296
00:13:42.715 --> 00:13:45.075
<00:13:42.715>to <00:13:42.870>say, <00:13:43.026>and <00:13:43.181>now <00:13:43.337>we <00:13:43.492>have <00:13:43.648>options <00:13:43.803>to <00:13:43.959>store <00:13:44.115>it.

297
00:13:45.535 --> 00:13:48.155
<00:13:45.535>If <00:13:45.662>you <00:13:45.789>are <00:13:45.916>thinking <00:13:46.044>in <00:13:46.171>a <00:13:46.298>cloud <00:13:46.425>environment, <00:13:46.553>you <00:13:46.680>have <00:13:46.807>a <00:13:46.935>VM

298
00:13:48.155 --> 00:13:49.475
<00:13:48.155>and <00:13:48.392>you <00:13:48.630>can <00:13:48.867>attach <00:13:49.105>block <00:13:49.343>storage.

299
00:13:50.375 --> 00:13:52.995
<00:13:50.375>So <00:13:50.575>this <00:13:50.775>is <00:13:50.975>kind <00:13:51.175>of <00:13:51.375>your <00:13:51.575>local <00:13:51.775>disc.

300
00:13:53.065 --> 00:13:54.435
<00:13:53.065>It's <00:13:53.311>not <00:13:53.558>entirely <00:13:53.804>a <00:13:54.051>local <00:13:54.298>DIS

301
00:13:54.435 --> 00:13:56.315
<00:13:54.435>because <00:13:54.635>you're <00:13:54.834>still <00:13:55.035>accessing <00:13:55.234>it <00:13:55.435>over <00:13:55.635>the <00:13:55.834>network.

302
00:13:56.935 --> 00:13:59.435
<00:13:56.935>Um, <00:13:57.074>we <00:13:57.214>have <00:13:57.354>virtual <00:13:57.494>machines <00:13:57.635>that <00:13:57.775>really <00:13:57.915>have <00:13:58.055>a <00:13:58.194>local <00:13:58.334>disk,

303
00:13:59.435 --> 00:14:01.955
<00:13:59.435>so <00:13:59.590>the <00:13:59.746>hypervisor <00:13:59.901>has <00:14:00.057>attached <00:14:00.212>some <00:14:00.368>real <00:14:00.523>storage <00:14:00.679>to <00:14:00.834>it.

304
00:14:03.585 --> 00:14:07.355
<00:14:03.585>Then <00:14:03.740>maybe <00:14:03.896>you <00:14:04.051>have <00:14:04.207>options <00:14:04.362>with <00:14:04.518>a <00:14:04.673>CS <00:14:04.829>CF <00:14:04.985>guard,

305
00:14:09.195 --> 00:14:10.555
<00:14:09.195>S3 <00:14:09.807>CSI <00:14:10.419>driver.

306
00:14:11.095 --> 00:14:14.075
<00:14:11.095>So <00:14:11.222>instead <00:14:11.349>of <00:14:11.476>downloading <00:14:11.604>from <00:14:11.731>is <00:14:11.858>free, <00:14:11.985>you <00:14:12.113>attach <00:14:12.240>it <00:14:12.367>as <00:14:12.495>a

307
00:14:14.945 --> 00:14:17.435
<00:14:14.945>container <00:14:15.145>storage <00:14:15.345>interface <00:14:15.545>and <00:14:15.745>download <00:14:15.945>it <00:14:16.145>from <00:14:16.345>there.

308
00:14:17.835 --> 00:14:19.555
<00:14:17.835>I <00:14:17.990>also <00:14:18.146>tried <00:14:18.301>it <00:14:18.457>out <00:14:18.612>and <00:14:18.768>it <00:14:18.923>failed <00:14:19.079>a <00:14:19.235>couple

309
00:14:19.555 --> 00:14:20.595
<00:14:19.555>of <00:14:20.023>times <00:14:20.491>unfortunately.

310
00:14:21.335 --> 00:14:22.955
<00:14:21.335>And <00:14:21.568>what <00:14:21.801>most <00:14:22.035>guys <00:14:22.268>are <00:14:22.501>doing <00:14:22.735>is

311
00:14:22.955 --> 00:14:24.195
<00:14:22.955>using <00:14:23.234>a <00:14:23.513>network <00:14:23.792>file <00:14:24.071>storage.

312
00:14:24.815 --> 00:14:28.555
<00:14:24.815>So <00:14:24.970>each <00:14:25.126>single <00:14:25.281>VM <00:14:25.437>gets <00:14:25.592>attached <00:14:25.748>this <00:14:25.903>network <00:14:26.059>file <00:14:26.215>storage

313
00:14:28.615 --> 00:14:30.635
<00:14:28.615>and <00:14:28.815>every <00:14:29.015>single <00:14:29.215>VM <00:14:29.415>can <00:14:29.615>read <00:14:29.815>from <00:14:30.015>there.

314
00:14:31.855 --> 00:14:34.315
<00:14:31.855>So <00:14:32.010>we <00:14:32.166>don't <00:14:32.321>have <00:14:32.477>to <00:14:32.632>re-download <00:14:32.788>it <00:14:32.943>from <00:14:33.099>hugging <00:14:33.255>face,

315
00:14:34.315 --> 00:14:37.595
<00:14:34.315>from <00:14:34.515>object <00:14:34.715>storage <00:14:34.915>from <00:14:35.115>CDNs, <00:14:35.315>so <00:14:35.515>also <00:14:35.715>whatever,

316
00:14:37.735 --> 00:14:39.835
<00:14:37.735>but <00:14:37.968>can <00:14:38.201>use <00:14:38.435>the <00:14:38.668>network <00:14:38.901>file <00:14:39.135>storage

317
00:14:40.495 --> 00:14:41.875
<00:14:40.495>and <00:14:40.743>attach <00:14:40.991>it <00:14:41.240>to <00:14:41.488>the <00:14:41.737>vm.

318
00:14:41.945 --> 00:14:44.115
<00:14:41.945>This <00:14:42.178>is <00:14:42.411>especially <00:14:42.645>important <00:14:42.878>if <00:14:43.111>we <00:14:43.345>want

319
00:14:44.115 --> 00:14:45.195
<00:14:44.115>to <00:14:44.601>scale <00:14:45.087>horizontally.

320
00:14:45.575 --> 00:14:47.315
<00:14:45.575>So <00:14:45.808>we <00:14:46.041>are <00:14:46.275>recognizing <00:14:46.508>there's <00:14:46.741>small <00:14:46.975>load.

321
00:14:47.335 --> 00:14:50.915
<00:14:47.335>We <00:14:47.451>want <00:14:47.568>to <00:14:47.685>spin <00:14:47.801>up <00:14:47.918>a <00:14:48.035>new <00:14:48.151>vm, <00:14:48.268>install <00:14:48.385>the <00:14:48.501>uh, <00:14:48.618>piping <00:14:48.735>script

322
00:14:51.225 --> 00:14:54.245
<00:14:51.225>and <00:14:51.425>download <00:14:51.625>the <00:14:51.825>model <00:14:52.025>weights <00:14:52.225>from <00:14:52.425>our <00:14:52.625>NFS.

323
00:14:54.515 --> 00:14:57.645
<00:14:54.515>This <00:14:54.670>is <00:14:54.826>optimizing <00:14:54.981>how <00:14:55.137>fast <00:14:55.292>we <00:14:55.448>can <00:14:55.603>serve <00:14:55.759>the <00:14:55.915>model

324
00:14:58.345 --> 00:15:01.085
<00:14:58.345>and <00:14:58.461>a <00:14:58.578>code <00:14:58.695>startup <00:14:58.811>time <00:14:58.928>where <00:14:59.045>the <00:14:59.161>model <00:14:59.278>is <00:14:59.395>not <00:14:59.511>in <00:14:59.628>some <00:14:59.745>kind

325
00:15:01.085 --> 00:15:04.365
<00:15:01.085>of <00:15:01.225>storage, <00:15:01.365>it <00:15:01.505>takes <00:15:01.645>up <00:15:01.785>to <00:15:01.925>20 <00:15:02.065>minutes <00:15:02.205>to <00:15:02.345>download <00:15:02.485>it

326
00:15:04.365 --> 00:15:06.685
<00:15:04.365>and <00:15:04.520>that's <00:15:04.676>not <00:15:04.831>acceptable <00:15:04.987>if <00:15:05.142>we <00:15:05.298>want <00:15:05.453>to <00:15:05.609>scale <00:15:05.765>horizontally,

327
00:15:06.685 --> 00:15:10.285
<00:15:06.685>like <00:15:06.824>spinning <00:15:06.964>up <00:15:07.104>new <00:15:07.244>VMs <00:15:07.385>very <00:15:07.525>fast <00:15:07.665>to <00:15:07.805>serve <00:15:07.944>more <00:15:08.084>requests,

328
00:15:10.555 --> 00:15:13.125
<00:15:10.555>therefore <00:15:10.729>we <00:15:10.905>are <00:15:11.079>attaching <00:15:11.255>uh, <00:15:11.430>this <00:15:11.604>network <00:15:11.780>file <00:15:11.954>storage.

329
00:15:15.025 --> 00:15:17.325
<00:15:15.025>Um, <00:15:15.305>let's <00:15:15.584>skip <00:15:15.865>through <00:15:16.145>the <00:15:16.425>points

330
00:15:22.805 --> 00:15:23.095
next.

331
00:15:23.095 --> 00:15:24.855
<00:15:23.095>Billing <00:15:23.795>is <00:15:24.495>kind

332
00:15:24.855 --> 00:15:28.335
<00:15:24.855>of <00:15:25.010>easy <00:15:25.166>I <00:15:25.321>think <00:15:25.477>in <00:15:25.632>any <00:15:25.788>source <00:15:25.943>environment <00:15:26.099>you <00:15:26.255>implemented

333
00:15:28.335 --> 00:15:29.455
<00:15:28.335>somewhere <00:15:28.839>like <00:15:29.343>this.

334
00:15:30.475 --> 00:15:31.655
<00:15:30.475>So <00:15:30.740>let's <00:15:31.006>go <00:15:31.271>through <00:15:31.537>it.

335
00:15:31.675 --> 00:15:34.575
<00:15:31.675>So <00:15:31.814>you, <00:15:31.954>the <00:15:32.094>user <00:15:32.234>sends <00:15:32.375>a <00:15:32.515>request, <00:15:32.655>it <00:15:32.795>selects <00:15:32.935>a <00:15:33.074>model

336
00:15:34.715 --> 00:15:36.375
<00:15:34.715>and <00:15:34.948>what <00:15:35.181>kind <00:15:35.415>of <00:15:35.648>prompt <00:15:35.881>he's <00:15:36.115>using.

337
00:15:38.285 --> 00:15:39.855
<00:15:38.285>Then <00:15:38.635>we <00:15:38.985>have <00:15:39.334>a <00:15:39.685>middleware.

338
00:15:40.045 --> 00:15:41.735
<00:15:40.045>It's <00:15:40.324>actually <00:15:40.604>kind <00:15:40.885>of <00:15:41.165>a <00:15:41.444>microservice

339
00:15:41.835 --> 00:15:44.015
<00:15:41.835>but <00:15:42.115>more <00:15:42.395>like <00:15:42.675>a <00:15:42.955>HDP <00:15:43.235>proxy.

340
00:15:44.605 --> 00:15:45.855
<00:15:44.605>This <00:15:44.886>takes <00:15:45.167>in <00:15:45.448>the <00:15:45.730>request

341
00:15:46.195 --> 00:15:48.055
<00:15:46.195>and <00:15:46.475>doing <00:15:46.755>some <00:15:47.035>kind <00:15:47.315>of <00:15:47.595>validation

342
00:15:48.645 --> 00:15:50.175
<00:15:48.645>rate <00:15:48.920>limiting <00:15:49.195>in <00:15:49.471>the <00:15:49.746>next <00:15:50.021>slides.

343
00:15:53.845 --> 00:15:55.505
<00:15:53.845>And <00:15:54.125>that's <00:15:54.405>basically <00:15:54.685>what <00:15:54.965>it <00:15:55.245>does.

344
00:15:55.535 --> 00:15:56.785
<00:15:55.535>It's <00:15:55.816>a <00:15:56.097>very <00:15:56.378>small <00:15:56.660>application.

345
00:15:56.855 --> 00:15:58.305
<00:15:56.855>Then <00:15:57.181>it <00:15:57.507>redirects <00:15:57.833>the <00:15:58.160>prompt

346
00:15:58.445 --> 00:16:02.185
<00:15:58.445>to <00:15:58.620>the <00:15:58.795>real <00:15:58.970>LLM <00:15:59.145>in <00:15:59.320>this <00:15:59.495>inference <00:15:59.670>engine <00:15:59.845>container.

347
00:16:04.005 --> 00:16:08.705
<00:16:04.005>And <00:16:04.180>then <00:16:04.355>the <00:16:04.530>LLM <00:16:04.705>answers <00:16:04.880>with <00:16:05.055>generated <00:16:05.230>text <00:16:05.405>obviously,

348
00:16:09.285 --> 00:16:11.385
<00:16:09.285>and <00:16:09.518>it <00:16:09.751>has <00:16:09.985>attribute <00:16:10.218>with <00:16:10.451>the <00:16:10.685>usage

349
00:16:11.845 --> 00:16:14.825
<00:16:11.845>and <00:16:12.000>the <00:16:12.156>future <00:16:12.311>usage <00:16:12.467>we <00:16:12.622>can <00:16:12.778>use <00:16:12.933>for <00:16:13.089>the <00:16:13.245>billing.

350
00:16:15.045 --> 00:16:18.705
<00:16:15.045>And <00:16:15.245>most <00:16:15.444>providers <00:16:15.645>are <00:16:15.844>doing <00:16:16.045>millions <00:16:16.245>of <00:16:16.444>tokens

351
00:16:20.135 --> 00:16:24.595
<00:16:20.135>as <00:16:20.415>a <00:16:20.694>concurrency <00:16:20.975>model, <00:16:21.255>currency <00:16:21.535>model.

352
00:16:26.575 --> 00:16:29.595
<00:16:26.575>And <00:16:26.730>in <00:16:26.886>our <00:16:27.041>system <00:16:27.197>you <00:16:27.352>can <00:16:27.508>just <00:16:27.663>put <00:16:27.819>push <00:16:27.975>it

353
00:16:29.615 --> 00:16:31.315
<00:16:29.615>to <00:16:29.965>uh, <00:16:30.315>a <00:16:30.665>payment <00:16:31.015>provider.

354
00:16:31.855 --> 00:16:35.875
<00:16:31.855>You <00:16:31.982>say, <00:16:32.109>okay, <00:16:32.236>this <00:16:32.364>user <00:16:32.491>has <00:16:32.618>uh, <00:16:32.745>used <00:16:32.873>one <00:16:33.000>uh, <00:16:33.127>from <00:16:33.255>token

355
00:16:35.875 --> 00:16:36.955
<00:16:35.875>and <00:16:36.199>one <00:16:36.523>completion <00:16:36.847>token.

356
00:16:37.455 --> 00:16:41.195
<00:16:37.455>But <00:16:37.655>sometimes <00:16:37.855>this <00:16:38.055>payment <00:16:38.255>provider <00:16:38.455>has <00:16:38.655>tied <00:16:38.855>rate,

357
00:16:41.195 --> 00:16:42.515
<00:16:41.195>missing <00:16:41.789>rate <00:16:42.383>limiting

358
00:16:43.295 --> 00:16:45.755
<00:16:43.295>and <00:16:43.574>then <00:16:43.854>you <00:16:44.135>optimize <00:16:44.415>it <00:16:44.694>with

359
00:16:48.135 --> 00:16:51.575
<00:16:48.135>aggregating <00:16:48.335>all <00:16:48.535>the <00:16:48.735>tokens <00:16:48.935>a <00:16:49.135>user <00:16:49.335>has <00:16:49.535>used.

360
00:16:52.195 --> 00:16:54.695
<00:16:52.195>We <00:16:52.395>are <00:16:52.595>using <00:16:52.795>Redis <00:16:52.995>for <00:16:53.195>this <00:16:53.395>aggregation <00:16:53.595>currently,

361
00:16:54.915 --> 00:16:56.735
<00:16:54.915>but <00:16:55.115>you <00:16:55.314>can <00:16:55.515>also <00:16:55.714>do <00:16:55.915>it <00:16:56.115>with <00:16:56.314>Kafka.

362
00:16:57.895 --> 00:17:01.705
<00:16:57.895>I <00:16:58.128>think <00:16:58.361>the <00:16:58.595>limits <00:16:58.828>are <00:16:59.061>not <00:16:59.295>there.

363
00:17:03.075 --> 00:17:04.575
<00:17:03.075>So <00:17:03.300>we <00:17:03.525>are <00:17:03.750>storing <00:17:03.975>all <00:17:04.200>the <00:17:04.425>tokens

364
00:17:04.575 --> 00:17:06.895
<00:17:04.575>of <00:17:04.750>user <00:17:04.925>has <00:17:05.100>used <00:17:05.275>in <00:17:05.450>a <00:17:05.625>one <00:17:05.800>hour <00:17:05.975>interval.

365
00:17:07.155 --> 00:17:08.455
<00:17:07.155>So <00:17:07.388>we <00:17:07.623>are <00:17:07.857>sending <00:17:08.090>one <00:17:08.324>request

366
00:17:08.455 --> 00:17:10.135
<00:17:08.454>to <00:17:08.734>our <00:17:09.014>payment <00:17:09.294>provider <00:17:09.574>per <00:17:09.855>hour

367
00:17:10.595 --> 00:17:12.935
<00:17:10.595>and <00:17:10.750>says, <00:17:10.906>okay, <00:17:11.061>this <00:17:11.217>user <00:17:11.372>has <00:17:11.528>used <00:17:11.683>2 <00:17:11.839>million <00:17:11.995>tokens,

368
00:17:12.935 --> 00:17:15.175
<00:17:12.935>3 <00:17:13.109>million <00:17:13.284>tokens <00:17:13.460>over <00:17:13.635>whatsoever <00:17:13.810>over <00:17:13.984>this <00:17:14.159>one <00:17:14.335>hour.

369
00:17:15.315 --> 00:17:17.055
<00:17:15.315>So <00:17:15.515>it's <00:17:15.715>not <00:17:15.915>really <00:17:16.115>broken <00:17:16.315>down <00:17:16.515>per <00:17:16.715>request,

370
00:17:17.155 --> 00:17:18.295
<00:17:17.155>but <00:17:17.497>on <00:17:17.839>hourly <00:17:18.181>base

371
00:17:18.405 --> 00:17:20.615
<00:17:18.405>because <00:17:18.638>we <00:17:18.871>had <00:17:19.105>to <00:17:19.338>optimize <00:17:19.571>for <00:17:19.805>our

372
00:17:21.805 --> 00:17:23.255
<00:17:21.805>payment <00:17:22.457>provider <00:17:23.110>basically.

373
00:17:24.465 --> 00:17:26.445
<00:17:24.464>And <00:17:24.620>this <00:17:24.776>is <00:17:24.931>something <00:17:25.087>you <00:17:25.242>can <00:17:25.398>do <00:17:25.553>with <00:17:25.709>every <00:17:25.865>single

374
00:17:27.185 --> 00:17:29.445
<00:17:27.185>SA <00:17:27.359>service <00:17:27.534>you <00:17:27.710>provide <00:17:27.885>in <00:17:28.060>a <00:17:28.234>paper <00:17:28.409>use <00:17:28.585>manner.

375
00:17:34.825 --> 00:17:35.985
<00:17:34.825>I <00:17:35.086>think <00:17:35.347>it's <00:17:35.608>very <00:17:35.869>interesting.

376
00:17:36.325 --> 00:17:38.585
<00:17:36.325>Um, <00:17:36.558>once <00:17:36.791>you <00:17:37.025>build <00:17:37.258>a <00:17:37.491>HDP <00:17:37.725>proxy,

377
00:17:38.785 --> 00:17:40.425
<00:17:38.785>I <00:17:38.940>think <00:17:39.096>you <00:17:39.251>can <00:17:39.407>build <00:17:39.562>it <00:17:39.718>in <00:17:39.873>every <00:17:40.029>single <00:17:40.185>language

378
00:17:40.445 --> 00:17:41.585
<00:17:40.445>and <00:17:40.787>it's <00:17:41.129>somehow <00:17:41.471>working.

379
00:17:42.685 --> 00:17:45.745
<00:17:42.685>But <00:17:42.801>in <00:17:42.918>this <00:17:43.034>LLM <00:17:43.151>context, <00:17:43.268>we <00:17:43.385>have <00:17:43.501>to <00:17:43.618>serve <00:17:43.735>a <00:17:43.851>lot <00:17:43.968>of <00:17:44.085>requests

380
00:17:46.285 --> 00:17:48.945
<00:17:46.285>and <00:17:46.751>small <00:17:47.218>optimizations <00:17:47.685>matter.

381
00:17:50.265 --> 00:17:53.365
<00:17:50.265>So <00:17:50.405>I <00:17:50.545>am <00:17:50.685>a <00:17:50.825>big <00:17:50.965>fan <00:17:51.105>of <00:17:51.245>Rust <00:17:51.385>and <00:17:51.525>Ping <00:17:51.665>Gora.

382
00:17:53.395 --> 00:17:56.525
<00:17:53.395>It's <00:17:53.628>a <00:17:53.861>framework <00:17:54.095>for <00:17:54.328>building <00:17:54.561>HGDP <00:17:54.795>process

383
00:17:56.585 --> 00:17:58.445
<00:17:56.585>and <00:17:56.818>low <00:17:57.051>balance <00:17:57.285>of <00:17:57.518>for <00:17:57.751>uh, <00:17:57.985>CloudFlare.

384
00:17:58.475 --> 00:18:03.005
<00:17:58.475>It's <00:17:58.615>really <00:17:58.754>battle <00:17:58.895>tested <00:17:59.034>and <00:17:59.175>I <00:17:59.314>have <00:17:59.455>run <00:17:59.594>a <00:17:59.735>small <00:17:59.875>benchmark.

385
00:18:03.145 --> 00:18:05.765
<00:18:03.145>So <00:18:03.272>I <00:18:03.399>have <00:18:03.526>implemented <00:18:03.654>a <00:18:03.781>proxy <00:18:03.908>with <00:18:04.035>Golan <00:18:04.163>and <00:18:04.290>with <00:18:04.417>P <00:18:04.545>Gora

386
00:18:05.905 --> 00:18:09.565
<00:18:05.905>and <00:18:06.080>at <00:18:06.254>the <00:18:06.430>end <00:18:06.605>of <00:18:06.780>the <00:18:06.955>day <00:18:07.129>we <00:18:07.305>observed

387
00:18:10.795 --> 00:18:14.285
<00:18:10.795>that <00:18:10.970>we <00:18:11.145>are <00:18:11.320>getting <00:18:11.495>a <00:18:11.670>240 <00:18:11.845>million <00:18:12.020>more <00:18:12.195>tokens

388
00:18:14.285 --> 00:18:17.005
<00:18:14.285>for <00:18:14.485>our <00:18:14.685>HDP <00:18:14.885>proxy <00:18:15.085>per <00:18:15.285>month <00:18:15.485>per <00:18:15.685>machine.

389
00:18:18.905 --> 00:18:21.365
<00:18:18.905>So <00:18:19.080>don't <00:18:19.254>reinvent <00:18:19.430>the <00:18:19.605>wheel, <00:18:19.780>just <00:18:19.955>use <00:18:20.129>the <00:18:20.305>library.

390
00:18:21.705 --> 00:18:25.605
<00:18:21.705>The <00:18:21.905>library <00:18:22.105>is <00:18:22.304>usually <00:18:22.504>way <00:18:22.705>better <00:18:22.905>optimiz <00:18:23.105>than

391
00:18:25.605 --> 00:18:28.925
<00:18:25.605>what <00:18:25.760>you <00:18:25.916>can <00:18:26.071>build <00:18:26.227>and <00:18:26.382>240 <00:18:26.538>million <00:18:26.693>tokens <00:18:26.849>was <00:18:27.005>insane

392
00:18:29.355 --> 00:18:32.565
<00:18:29.355>because <00:18:29.510>there's <00:18:29.666>like <00:18:29.821>half <00:18:29.977>of <00:18:30.132>the <00:18:30.288>cost <00:18:30.443>of <00:18:30.599>the <00:18:30.755>machine.

393
00:18:32.565 --> 00:18:37.165
<00:18:32.565>The <00:18:32.720>model <00:18:32.876>is <00:18:33.031>running, <00:18:33.187>obviously <00:18:33.342>using <00:18:33.498>ING <00:18:33.653>for <00:18:33.809>broker <00:18:33.965>pool.

394
00:18:37.215 --> 00:18:39.965
<00:18:37.215>Don't <00:18:37.495>spa <00:18:37.774>a <00:18:38.054>threatt <00:18:38.334>per <00:18:38.615>request.

395
00:18:39.965 --> 00:18:41.165
<00:18:39.965>You <00:18:40.181>send <00:18:40.397>to <00:18:40.612>your <00:18:40.829>premium <00:18:41.045>provider

396
00:18:41.225 --> 00:18:42.845
<00:18:41.225>or <00:18:41.458>to <00:18:41.691>this <00:18:41.925>aggregation <00:18:42.158>tool <00:18:42.391>like <00:18:42.625>red

397
00:18:45.005 --> 00:18:46.205
<00:18:45.005>Validate <00:18:45.275>inputs <00:18:45.545>early <00:18:45.815>and <00:18:46.085>fast.

398
00:18:47.145 --> 00:18:48.805
<00:18:47.145>Um, <00:18:47.425>that's <00:18:47.705>what <00:18:47.985>I <00:18:48.264>already <00:18:48.545>mentioned.

399
00:18:48.865 --> 00:18:52.645
<00:18:48.865>That's <00:18:49.098>one <00:18:49.331>functionality <00:18:49.565>of <00:18:49.798>this <00:18:50.031>token <00:18:50.265>middleware

400
00:18:52.875 --> 00:18:55.405
<00:18:52.875>because <00:18:53.075>once <00:18:53.275>we <00:18:53.475>interacted <00:18:53.675>with <00:18:53.875>this <00:18:54.075>inference <00:18:54.275>engine,

401
00:18:55.405 --> 00:18:57.445
<00:18:55.405>sometimes <00:18:55.638>it's <00:18:55.871>giving <00:18:56.105>back <00:18:56.338>weird <00:18:56.571>error <00:18:56.805>messages.

402
00:18:57.445 --> 00:18:59.965
<00:18:57.445>We <00:18:57.585>don't <00:18:57.725>really <00:18:57.865>want <00:18:58.004>to <00:18:58.145>have <00:18:58.284>based <00:18:58.425>on <00:18:58.564>the <00:18:58.705>user <00:18:58.845>input.

403
00:18:59.965 --> 00:19:02.045
<00:18:59.965>So <00:19:00.139>we <00:19:00.314>can <00:19:00.490>uh, <00:19:00.665>validate <00:19:00.840>the <00:19:01.014>user <00:19:01.189>input <00:19:01.365>before

404
00:19:02.105 --> 00:19:03.885
<00:19:02.105>and <00:19:02.305>then <00:19:02.505>proxy <00:19:02.705>it <00:19:02.905>to <00:19:03.105>the <00:19:03.305>inference <00:19:03.505>engine.

405
00:19:03.905 --> 00:19:06.165
<00:19:03.905>So <00:19:04.138>it <00:19:04.371>does <00:19:04.605>not <00:19:04.838>do <00:19:05.071>any <00:19:05.305>shenanigans.

406
00:19:08.335 --> 00:19:10.965
<00:19:08.335>Don't <00:19:08.615>don't <00:19:08.895>handcraft <00:19:09.175>re <00:19:09.455>request <00:19:09.735>passing.

407
00:19:12.265 --> 00:19:15.925
<00:19:12.265>So <00:19:12.440>the <00:19:12.615>open <00:19:12.790>AI <00:19:12.965>API <00:19:13.140>is <00:19:13.315>very <00:19:13.490>well <00:19:13.665>specified.

408
00:19:16.025 --> 00:19:18.485
<00:19:16.025>You <00:19:16.258>can <00:19:16.491>just <00:19:16.725>download <00:19:16.958>the <00:19:17.191>uh, <00:19:17.425>specification,

409
00:19:18.515 --> 00:19:19.525
<00:19:18.515>then <00:19:18.818>generate <00:19:19.121>the <00:19:19.424>models

410
00:19:19.945 --> 00:19:23.205
<00:19:19.945>and <00:19:20.100>use <00:19:20.256>it <00:19:20.411>inside <00:19:20.567>the <00:19:20.722>code <00:19:20.878>to <00:19:21.033>validate <00:19:21.189>the <00:19:21.345>request.

411
00:19:23.945 --> 00:19:26.045
<00:19:23.945>So <00:19:24.225>don't <00:19:24.504>read <00:19:24.784>through <00:19:25.064>the <00:19:25.345>documentation,

412
00:19:28.005 --> 00:19:30.705
<00:19:28.005>but <00:19:28.180>rather <00:19:28.355>use <00:19:28.530>the <00:19:28.705>documentation <00:19:28.880>to <00:19:29.055>generate <00:19:29.230>a <00:19:29.405>code,

413
00:19:31.655 --> 00:19:35.225
<00:19:31.655>obviously <00:19:31.855>benchmark <00:19:32.055>and <00:19:32.254>be <00:19:32.455>mindful <00:19:32.655>of <00:19:32.855>the <00:19:33.055>latency.

414
00:19:35.725 --> 00:19:38.985
<00:19:35.725>So <00:19:35.865>we <00:19:36.004>also <00:19:36.145>have <00:19:36.284>evaluated <00:19:36.425>if <00:19:36.564>we <00:19:36.705>can <00:19:36.844>stick <00:19:36.985>this <00:19:37.125>token

415
00:19:38.985 --> 00:19:41.865
<00:19:38.985>middleware <00:19:39.185>inside <00:19:39.385>the <00:19:39.584>inference <00:19:39.784>engine <00:19:39.985>as <00:19:40.185>a <00:19:40.385>plugin

416
00:19:42.175 --> 00:19:45.225
<00:19:42.175>because <00:19:42.375>every <00:19:42.575>single <00:19:42.774>network <00:19:42.975>request <00:19:43.175>costs <00:19:43.375>you <00:19:43.575>latency

417
00:19:45.225 --> 00:19:48.505
<00:19:45.225>of <00:19:45.365>course, <00:19:45.504>and <00:19:45.645>if <00:19:45.784>it's <00:19:45.925>in <00:19:46.064>memory, <00:19:46.205>it's <00:19:46.344>way, <00:19:46.485>way <00:19:46.625>faster.

418
00:19:49.645 --> 00:19:52.425
<00:19:49.645>But <00:19:49.820>still <00:19:49.995>we <00:19:50.170>want <00:19:50.345>to <00:19:50.520>be <00:19:50.695>highly <00:19:50.870>uh, <00:19:51.045>pluggable.

419
00:19:52.435 --> 00:19:54.905
<00:19:52.435>Maybe <00:19:52.668>some <00:19:52.901>day <00:19:53.135>we <00:19:53.368>recognize, <00:19:53.601>okay, <00:19:53.835>VLM

420
00:19:54.905 --> 00:19:56.065
<00:19:54.905>as <00:19:55.054>an <00:19:55.203>inference <00:19:55.352>engine <00:19:55.501>is <00:19:55.650>not <00:19:55.799>so <00:19:55.949>great,

421
00:19:56.175 --> 00:19:57.505
<00:19:56.175>then <00:19:56.414>we <00:19:56.653>can <00:19:56.893>just <00:19:57.132>get <00:19:57.372>out

422
00:19:58.405 --> 00:20:00.465
<00:19:58.405>and <00:19:58.605>are <00:19:58.805>not <00:19:59.004>dependent <00:19:59.205>with <00:19:59.405>our <00:19:59.605>own <00:19:59.805>code.

423
00:20:02.645 --> 00:20:07.195
<00:20:02.645>All <00:20:02.785>right, <00:20:02.925>I <00:20:03.065>think <00:20:03.205>rate <00:20:03.345>limit <00:20:03.485>is <00:20:03.625>really <00:20:03.764>interesting <00:20:03.905>in <00:20:04.045>the

424
00:20:07.195 --> 00:20:08.195
<00:20:07.195>context <00:20:07.645>of <00:20:08.095>LLMs.

425
00:20:09.815 --> 00:20:11.835
<00:20:09.815>So <00:20:09.970>I <00:20:10.126>think <00:20:10.281>there <00:20:10.437>are <00:20:10.592>two <00:20:10.748>kinds <00:20:10.903>of <00:20:11.059>rate <00:20:11.215>limitings.

426
00:20:12.015 --> 00:20:13.955
<00:20:12.015>The <00:20:12.215>most <00:20:12.415>obvious <00:20:12.615>one <00:20:12.815>is <00:20:13.015>request <00:20:13.215>rate <00:20:13.415>limiting,

427
00:20:13.955 --> 00:20:15.555
<00:20:13.955>which <00:20:14.235>you're <00:20:14.514>doing <00:20:14.794>on <00:20:15.074>all <00:20:15.355>kinds

428
00:20:15.555 --> 00:20:16.995
<00:20:15.555>of <00:20:15.814>SA <00:20:16.073>solutions <00:20:16.332>you <00:20:16.591>are <00:20:16.851>building.

429
00:20:17.455 --> 00:20:19.675
<00:20:17.455>So <00:20:17.655>a <00:20:17.855>user <00:20:18.054>sensor <00:20:18.254>request, <00:20:18.455>then <00:20:18.655>it <00:20:18.855>goes

430
00:20:19.675 --> 00:20:21.595
<00:20:19.675>through <00:20:19.955>the <00:20:20.235>request <00:20:20.514>rate <00:20:20.794>limiting <00:20:21.075>component.

431
00:20:21.825 --> 00:20:23.635
<00:20:21.825>Usually <00:20:22.291>it's <00:20:22.758>around <00:20:23.225>Redis

432
00:20:23.655 --> 00:20:26.435
<00:20:23.655>and <00:20:23.888>a <00:20:24.121>token <00:20:24.355>bucket, <00:20:24.588>leaky <00:20:24.821>bucket <00:20:25.055>algorithm.

433
00:20:27.585 --> 00:20:31.435
<00:20:27.585>Then <00:20:27.725>we <00:20:27.865>are <00:20:28.005>taking <00:20:28.145>out <00:20:28.285>one <00:20:28.425>token <00:20:28.565>of <00:20:28.705>this <00:20:28.845>token <00:20:28.985>bucket

434
00:20:31.655 --> 00:20:33.235
<00:20:31.655>and <00:20:32.004>have <00:20:32.355>the <00:20:32.705>limit <00:20:33.055>left.

435
00:20:33.735 --> 00:20:35.315
<00:20:33.735>That's <00:20:34.014>very, <00:20:34.294>very <00:20:34.574>easy <00:20:34.854>to <00:20:35.135>implement

436
00:20:35.335 --> 00:20:37.595
<00:20:35.335>and <00:20:35.490>we <00:20:35.646>did <00:20:35.801>it <00:20:35.957>in <00:20:36.112>like <00:20:36.268>10 <00:20:36.423>minutes <00:20:36.579>or <00:20:36.735>something.

437
00:20:38.135 --> 00:20:41.955
<00:20:38.135>But <00:20:38.275>now <00:20:38.415>there's <00:20:38.555>also <00:20:38.695>another <00:20:38.835>component <00:20:38.975>we <00:20:39.115>need <00:20:39.254>to <00:20:39.395>rate <00:20:39.535>limit

438
00:20:42.135 --> 00:20:44.755
<00:20:42.135>and <00:20:42.310>this <00:20:42.485>is <00:20:42.660>the <00:20:42.835>token <00:20:43.010>used <00:20:43.185>for <00:20:43.360>a <00:20:43.535>user

439
00:20:45.225 --> 00:20:49.885
<00:20:45.225>because <00:20:45.341>a <00:20:45.458>user <00:20:45.574>can <00:20:45.691>send <00:20:45.808>a <00:20:45.925>request <00:20:46.041>like <00:20:46.158>write <00:20:46.274>me <00:20:46.391>a <00:20:46.508>book <00:20:46.625>of

440
00:20:50.645 --> 00:20:53.885
<00:20:50.645>consensus <00:20:50.820>algorithms <00:20:50.995>and <00:20:51.170>then <00:20:51.345>the <00:20:51.520>model <00:20:51.695>is <00:20:51.870>busy <00:20:52.045>generating

441
00:20:54.065 --> 00:20:56.405
<00:20:54.065>all <00:20:54.298>the <00:20:54.531>uh, <00:20:54.765>text <00:20:54.998>for <00:20:55.231>this <00:20:55.465>book.

442
00:20:57.445 --> 00:20:59.305
<00:20:57.445>And <00:20:57.725>we <00:20:58.004>basically <00:20:58.284>want <00:20:58.564>to <00:20:58.845>avoid

443
00:20:59.305 --> 00:21:02.385
<00:20:59.305>that <00:20:59.480>a <00:20:59.655>single <00:20:59.830>user <00:21:00.005>is <00:21:00.180>sending <00:21:00.355>this <00:21:00.530>huge <00:21:00.705>bronze

444
00:21:02.445 --> 00:21:03.785
<00:21:02.445>and <00:21:02.617>the <00:21:02.789>model <00:21:02.961>is <00:21:03.134>busy <00:21:03.306>all <00:21:03.478>the <00:21:03.651>time

445
00:21:05.845 --> 00:21:08.095
<00:21:05.845>because <00:21:06.020>it's <00:21:06.195>a, <00:21:06.370>if <00:21:06.545>it's <00:21:06.720>busy <00:21:06.895>for <00:21:07.070>one <00:21:07.245>user,

446
00:21:08.095 --> 00:21:09.935
<00:21:08.095>another <00:21:08.375>user <00:21:08.655>can't <00:21:08.935>use <00:21:09.215>it <00:21:09.495>anymore.

447
00:21:11.315 --> 00:21:15.055
<00:21:11.315>So <00:21:11.470>we <00:21:11.626>also <00:21:11.781>implemented <00:21:11.937>a <00:21:12.092>request <00:21:12.248>limit <00:21:12.403>on <00:21:12.559>token <00:21:12.715>level.

448
00:21:15.765 --> 00:21:18.695
<00:21:15.765>Unfortunately <00:21:16.115>it's <00:21:16.465>not <00:21:16.815>super <00:21:17.165>straightforward

449
00:21:19.165 --> 00:21:22.575
<00:21:19.165>because <00:21:19.514>the <00:21:19.865>tokens <00:21:20.215>are <00:21:20.565>splitted

450
00:21:22.575 --> 00:21:24.495
<00:21:22.575>between <00:21:22.855>prompt <00:21:23.135>tokens <00:21:23.415>and <00:21:23.695>output <00:21:23.975>tokens.

451
00:21:24.755 --> 00:21:27.895
<00:21:24.755>The <00:21:24.930>output <00:21:25.105>tokens <00:21:25.280>are <00:21:25.455>generally <00:21:25.630>after <00:21:25.805>the <00:21:25.980>model <00:21:26.155>answered.

452
00:21:28.195 --> 00:21:29.895
<00:21:28.195>So <00:21:28.395>we <00:21:28.595>can <00:21:28.794>really <00:21:28.995>write <00:21:29.195>limit <00:21:29.395>on <00:21:29.595>that.

453
00:21:29.915 --> 00:21:33.175
<00:21:29.915>So <00:21:30.055>we <00:21:30.195>can <00:21:30.335>only <00:21:30.475>write <00:21:30.615>limit <00:21:30.754>on <00:21:30.895>the <00:21:31.034>uh, <00:21:31.175>prompt <00:21:31.315>tokens,

454
00:21:35.075 --> 00:21:36.215
<00:21:35.075>but <00:21:35.280>still <00:21:35.485>this <00:21:35.690>is <00:21:35.895>not <00:21:36.101>enough.

455
00:21:36.235 --> 00:21:38.095
<00:21:36.235>So <00:21:36.409>if <00:21:36.584>a <00:21:36.760>user <00:21:36.935>has <00:21:37.110>a <00:21:37.284>really <00:21:37.459>short <00:21:37.635>prompt,

456
00:21:38.225 --> 00:21:40.055
<00:21:38.225>which <00:21:38.458>generates <00:21:38.691>a <00:21:38.925>lot <00:21:39.158>of <00:21:39.391>output <00:21:39.625>tokens,

457
00:21:40.075 --> 00:21:42.535
<00:21:40.075>the <00:21:40.355>machine <00:21:40.635>is <00:21:40.915>really, <00:21:41.195>really <00:21:41.475>busy

458
00:21:42.835 --> 00:21:47.015
<00:21:42.835>and <00:21:43.035>a <00:21:43.235>single <00:21:43.435>user <00:21:43.635>can <00:21:43.835>overload <00:21:44.035>our <00:21:44.235>Machine

459
00:21:47.015 --> 00:21:49.295
<00:21:47.015>where <00:21:47.155>the <00:21:47.295>LLM <00:21:47.435>is <00:21:47.575>running <00:21:47.715>and <00:21:47.855>we <00:21:47.995>want <00:21:48.135>to <00:21:48.275>avoid <00:21:48.415>it.

460
00:21:49.355 --> 00:21:51.855
<00:21:49.355>So <00:21:49.495>we <00:21:49.635>added <00:21:49.775>an <00:21:49.915>estimate <00:21:50.055>for <00:21:50.195>the <00:21:50.335>output <00:21:50.475>tokens <00:21:50.615>as <00:21:50.755>well.

461
00:21:52.715 --> 00:21:55.935
<00:21:52.715>And <00:21:52.948>so <00:21:53.181>we <00:21:53.415>avoided <00:21:53.648>the <00:21:53.881>single <00:21:54.115>user

462
00:21:56.595 --> 00:21:58.135
<00:21:56.595>can <00:21:56.941>exhaust <00:21:57.288>our <00:21:57.634>entire <00:21:57.981>system

463
00:21:58.555 --> 00:22:02.015
<00:21:58.555>and <00:21:58.695>make <00:21:58.835>it <00:21:58.975>unavailable <00:21:59.115>for <00:21:59.255>all <00:21:59.395>the <00:21:59.535>other <00:21:59.675>users <00:21:59.815>we <00:21:59.955>have.

464
00:22:03.595 --> 00:22:06.335
<00:22:03.595>So <00:22:03.770>this <00:22:03.945>is <00:22:04.120>basically <00:22:04.295>the <00:22:04.470>entire <00:22:04.645>magic <00:22:04.820>around <00:22:04.995>it

465
00:22:07.035 --> 00:22:08.815
<00:22:07.035>and <00:22:07.210>we <00:22:07.385>have <00:22:07.560>kind <00:22:07.735>of <00:22:07.910>a <00:22:08.085>two <00:22:08.260>phase <00:22:08.435>mechanism.

466
00:22:09.955 --> 00:22:13.435
<00:22:09.955>The <00:22:10.304>first <00:22:10.655>step <00:22:11.004>is <00:22:11.355>estimating.

467
00:22:13.855 --> 00:22:16.675
<00:22:13.855>So <00:22:13.995>pro <00:22:14.135>tos <00:22:14.275>can <00:22:14.415>be <00:22:14.555>estimated <00:22:14.695>really <00:22:14.835>well <00:22:14.975>upper <00:22:15.115>tos <00:22:15.255>not

468
00:22:17.305 --> 00:22:19.675
<00:22:17.305>therefore <00:22:17.505>we <00:22:17.705>need <00:22:17.905>to <00:22:18.105>adjust <00:22:18.305>the <00:22:18.505>real <00:22:18.705>usage.

469
00:22:19.815 --> 00:22:22.835
<00:22:19.815>So <00:22:19.970>a <00:22:20.126>user <00:22:20.281>is <00:22:20.437>always <00:22:20.592>getting <00:22:20.748>what <00:22:20.903>he's <00:22:21.059>paying <00:22:21.215>for

470
00:22:24.055 --> 00:22:25.515
<00:22:24.055>and <00:22:24.274>to <00:22:24.493>adjust <00:22:24.712>for <00:22:24.931>the <00:22:25.150>real <00:22:25.369>use

471
00:22:25.515 --> 00:22:28.275
<00:22:25.515>as <00:22:25.690>we <00:22:25.865>are <00:22:26.040>still <00:22:26.215>using <00:22:26.390>the <00:22:26.565>attribute <00:22:26.740>usage <00:22:26.915>again.

472
00:22:29.255 --> 00:22:32.315
<00:22:29.255>So <00:22:29.410>it's <00:22:29.566>kind <00:22:29.721>of <00:22:29.877>a <00:22:30.032>transactional <00:22:30.188>build <00:22:30.343>and <00:22:30.499>to <00:22:30.655>Redis.

473
00:22:38.695 --> 00:22:40.425
<00:22:38.695>Yeah, <00:22:38.975>token <00:22:39.254>limits <00:22:39.534>must <00:22:39.814>be <00:22:40.095>enforced

474
00:22:40.975 --> 00:22:42.425
<00:22:40.975>with <00:22:41.301>some <00:22:41.627>kind <00:22:41.953>of <00:22:42.280>tricks.

475
00:22:42.845 --> 00:22:45.185
<00:22:42.845>We <00:22:43.000>don't <00:22:43.156>really, <00:22:43.311>we <00:22:43.467>don't <00:22:43.622>really <00:22:43.778>know <00:22:43.933>the <00:22:44.089>real <00:22:44.245>usage

476
00:22:45.315 --> 00:22:46.545
<00:22:45.315>until <00:22:45.499>we <00:22:45.684>send <00:22:45.868>it <00:22:46.053>to <00:22:46.237>the <00:22:46.422>alarm.

477
00:22:46.565 --> 00:22:49.065
<00:22:46.565>So <00:22:46.765>we <00:22:46.965>have <00:22:47.165>to <00:22:47.365>do <00:22:47.565>some <00:22:47.765>good <00:22:47.965>estimations

478
00:22:50.835 --> 00:22:52.505
<00:22:50.835>alarm <00:22:51.301>responses <00:22:51.768>take <00:22:52.235>time.

479
00:22:53.085 --> 00:22:54.465
<00:22:53.085>So <00:22:53.395>what <00:22:53.706>we <00:22:54.016>basically <00:22:54.327>want

480
00:22:54.465 --> 00:22:57.025
<00:22:54.465>to <00:22:54.620>avoid <00:22:54.776>is <00:22:54.931>a <00:22:55.087>user <00:22:55.242>send <00:22:55.398>a <00:22:55.553>hundred <00:22:55.709>request <00:22:55.865>which

481
00:22:57.105 --> 00:22:58.345
<00:22:57.105>fulfills <00:22:57.477>the <00:22:57.849>request <00:22:58.221>limit.

482
00:22:58.725 --> 00:23:00.985
<00:22:58.725>We <00:22:58.880>write <00:22:59.036>me <00:22:59.191>a <00:22:59.347>book <00:22:59.502>about <00:22:59.658>something <00:22:59.813>which <00:22:59.969>is <00:23:00.125>then

483
00:23:00.985 --> 00:23:02.145
<00:23:00.985>exhausting <00:23:01.332>the <00:23:01.681>entire <00:23:02.029>system.

484
00:23:03.085 --> 00:23:07.065
<00:23:03.085>Um, <00:23:03.225>and <00:23:03.365>to <00:23:03.505>avoid <00:23:03.645>that, <00:23:03.785>we <00:23:03.925>are <00:23:04.065>introducing <00:23:04.205>the <00:23:04.345>token <00:23:04.485>limit

485
00:23:08.315 --> 00:23:11.005
<00:23:08.315>With <00:23:08.490>the <00:23:08.665>rough <00:23:08.840>estimations, <00:23:09.015>which <00:23:09.190>are <00:23:09.365>probably <00:23:09.540>not <00:23:09.715>correct.

486
00:23:11.115 --> 00:23:13.605
<00:23:11.115>Therefore <00:23:11.290>we <00:23:11.465>are <00:23:11.640>doing <00:23:11.815>the <00:23:11.990>adjustment <00:23:12.165>at <00:23:12.340>the <00:23:12.515>end

487
00:23:13.605 --> 00:23:14.645
<00:23:13.605>of <00:23:14.073>the <00:23:14.541>request.

488
00:23:16.585 --> 00:23:19.485
<00:23:16.585>So <00:23:16.818>if <00:23:17.051>you're <00:23:17.285>interacting <00:23:17.518>with <00:23:17.751>open <00:23:17.985>ai,

489
00:23:19.985 --> 00:23:23.285
<00:23:19.985>you're <00:23:20.125>generating <00:23:20.264>API <00:23:20.405>keys, <00:23:20.544>you <00:23:20.685>have <00:23:20.824>seen <00:23:20.965>it, <00:23:21.104>it's <00:23:21.245>called <00:23:21.385>SK

490
00:23:23.345 --> 00:23:24.805
<00:23:23.345>and <00:23:23.783>some <00:23:24.221>random <00:23:24.659>UUID,

491
00:23:25.945 --> 00:23:28.685
<00:23:25.945>but <00:23:26.100>maybe <00:23:26.256>there's <00:23:26.411>more <00:23:26.567>which <00:23:26.722>we <00:23:26.878>can <00:23:27.033>use <00:23:27.189>for <00:23:27.345>authentication.

492
00:23:29.505 --> 00:23:32.205
<00:23:29.505>Um, <00:23:29.660>commonly <00:23:29.816>in <00:23:29.971>the <00:23:30.127>web <00:23:30.282>world <00:23:30.438>you <00:23:30.593>use <00:23:30.749>barrier <00:23:30.905>tokens,

493
00:23:32.205 --> 00:23:33.965
<00:23:32.205>you <00:23:32.438>can <00:23:32.671>encode <00:23:32.905>some <00:23:33.138>information <00:23:33.371>into <00:23:33.605>them,

494
00:23:34.265 --> 00:23:38.085
<00:23:34.265>you <00:23:34.405>can <00:23:34.545>add <00:23:34.685>more <00:23:34.825>attributes <00:23:34.965>like <00:23:35.105>an <00:23:35.245>expiry, <00:23:35.385>you <00:23:35.525>can <00:23:35.665>have

495
00:23:39.305 --> 00:23:41.525
<00:23:39.305>any <00:23:39.505>sort <00:23:39.705>of <00:23:39.905>information, <00:23:40.105>which <00:23:40.305>is <00:23:40.505>really <00:23:40.705>cool.

496
00:23:41.705 --> 00:23:44.205
<00:23:41.705>The <00:23:41.879>next <00:23:42.054>options <00:23:42.230>are <00:23:42.405>API <00:23:42.580>keys. <00:23:42.754>It's <00:23:42.929>very <00:23:43.105>simple.

497
00:23:44.265 --> 00:23:46.605
<00:23:44.265>You <00:23:44.392>can <00:23:44.519>look <00:23:44.646>up <00:23:44.774>into <00:23:44.901>the <00:23:45.028>database, <00:23:45.155>do <00:23:45.283>they <00:23:45.410>exist <00:23:45.537>or <00:23:45.665>not,

498
00:23:46.625 --> 00:23:50.405
<00:23:46.625>and <00:23:46.858>say, <00:23:47.091>yes, <00:23:47.325>amazing <00:23:47.558>use <00:23:47.791>authenticated <00:23:48.025>exists

499
00:23:50.585 --> 00:23:52.085
<00:23:50.585>and <00:23:50.922>if <00:23:51.260>not, <00:23:51.597>then <00:23:51.935>not.

500
00:23:53.275 --> 00:23:55.085
<00:23:53.275>What <00:23:53.508>I <00:23:53.741>really <00:23:53.975>like <00:23:54.208>is <00:23:54.441>mutual <00:23:54.675>TLS.

501
00:23:55.425 --> 00:23:56.925
<00:23:55.425>So <00:23:55.649>you <00:23:55.875>give <00:23:56.100>the <00:23:56.325>client <00:23:56.550>a <00:23:56.774>certificate

502
00:23:56.925 --> 00:23:58.045
<00:23:56.925>and <00:23:57.261>a <00:23:57.597>server <00:23:57.933>certificate

503
00:23:58.045 --> 00:24:01.005
<00:23:58.045>and <00:23:58.220>they <00:23:58.395>each <00:23:58.570>validating <00:23:58.745>if <00:23:58.920>the <00:23:59.095>certificate <00:23:59.270>is <00:23:59.445>valid.

504
00:24:01.605 --> 00:24:02.605
<00:24:01.605>I <00:24:01.830>think <00:24:02.055>it's <00:24:02.280>very <00:24:02.505>secure

505
00:24:03.465 --> 00:24:07.245
<00:24:03.465>and <00:24:03.592>tools <00:24:03.719>like <00:24:03.846>Istio, <00:24:03.974>a <00:24:04.101>service <00:24:04.228>mesh <00:24:04.355>is <00:24:04.483>doing <00:24:04.610>it <00:24:04.737>with <00:24:04.865>their

506
00:24:07.835 --> 00:24:10.005
<00:24:07.835>invoice <00:24:08.115>side <00:24:08.395>cards <00:24:08.675>between <00:24:08.955>the <00:24:09.235>components.

507
00:24:11.345 --> 00:24:15.285
<00:24:11.345>Our <00:24:11.500>requirements <00:24:11.656>the <00:24:11.811>are <00:24:11.967>the <00:24:12.122>credentials <00:24:12.278>must <00:24:12.433>be <00:24:12.589>long <00:24:12.745>lift.

508
00:24:15.755 --> 00:24:18.605
<00:24:15.755>Typically <00:24:16.035>our <00:24:16.315>customers <00:24:16.595>are <00:24:16.875>AI <00:24:17.155>engineers

509
00:24:18.605 --> 00:24:20.285
<00:24:18.605>and <00:24:18.885>they <00:24:19.165>really <00:24:19.445>have <00:24:19.725>credential <00:24:20.005>rotation.

510
00:24:21.265 --> 00:24:24.965
<00:24:21.265>So <00:24:21.498>we <00:24:21.731>want <00:24:21.965>to <00:24:22.198>give <00:24:22.431>out <00:24:22.665>credentials

511
00:24:24.965 --> 00:24:29.965
<00:24:24.965>that <00:24:25.245>are <00:24:25.524>living <00:24:25.804>forever <00:24:26.084>In <00:24:26.365>barrier

512
00:24:30.445 --> 00:24:33.165
<00:24:30.445>We <00:24:30.585>can <00:24:30.725>basically <00:24:30.865>do <00:24:31.004>that <00:24:31.145>by <00:24:31.284>setting <00:24:31.425>the <00:24:31.564>expiry <00:24:31.705>to, <00:24:31.845>uh,

513
00:24:33.165 --> 00:24:35.005
<00:24:33.165>the <00:24:33.865>year <00:24:34.565>9,999.

514
00:24:35.865 --> 00:24:38.605
<00:24:35.865>That's <00:24:36.040>really <00:24:36.215>discouraged. <00:24:36.390>You <00:24:36.565>shouldn't <00:24:36.740>do <00:24:36.915>that <00:24:37.090>at <00:24:37.265>all.

515
00:24:39.115 --> 00:24:41.605
<00:24:39.115>Usually <00:24:39.315>barrier <00:24:39.515>tokens <00:24:39.715>should <00:24:39.915>be, <00:24:40.115>uh, <00:24:40.315>short <00:24:40.515>lived

516
00:24:41.875 --> 00:24:43.685
<00:24:41.875>with <00:24:42.108>API <00:24:42.341>keys, <00:24:42.575>no <00:24:42.808>problem <00:24:43.041>at <00:24:43.275>all.

517
00:24:43.865 --> 00:24:46.845
<00:24:43.865>Mutual <00:24:44.215>DL <00:24:44.565>TLS <00:24:44.915>kind <00:24:45.265>of

518
00:24:48.405 --> 00:24:50.215
<00:24:48.405>then <00:24:48.685>you <00:24:48.965>want <00:24:49.245>to <00:24:49.524>support <00:24:49.805>TTL

519
00:24:50.215 --> 00:24:55.015
<00:24:50.215>because <00:24:50.448>some <00:24:50.681>customers <00:24:50.915>are <00:24:51.148>not <00:24:51.381>super, <00:24:51.615>um,

520
00:24:58.815 --> 00:25:01.065
<00:24:58.815>they <00:24:58.970>are <00:24:59.126>not <00:24:59.281>allowed <00:24:59.437>to <00:24:59.592>have <00:24:59.748>the <00:24:59.903>strong <00:25:00.059>living <00:25:00.215>credentials.

521
00:25:01.065 --> 00:25:03.905
<00:25:01.065>They <00:25:01.240>have <00:25:01.415>a <00:25:01.590>security <00:25:01.765>department <00:25:01.940>as <00:25:02.115>every <00:25:02.290>six <00:25:02.465>months.

522
00:25:04.045 --> 00:25:05.945
<00:25:04.045>We <00:25:04.220>need <00:25:04.395>to <00:25:04.570>rotate <00:25:04.745>them <00:25:04.920>to <00:25:05.095>be <00:25:05.270>more <00:25:05.445>secure.

523
00:25:06.405 --> 00:25:10.065
<00:25:06.405>So <00:25:06.545>we <00:25:06.685>want <00:25:06.825>to <00:25:06.965>build <00:25:07.105>in <00:25:07.245>a <00:25:07.385>mechanism <00:25:07.524>to <00:25:07.665>allow <00:25:07.805>them

524
00:25:10.245 --> 00:25:11.905
<00:25:10.245>to <00:25:10.594>set <00:25:10.945>this <00:25:11.294>TTL <00:25:11.645>timestamp.

525
00:25:12.255 --> 00:25:13.945
<00:25:12.255>With <00:25:12.535>bar <00:25:12.815>tokens, <00:25:13.095>it's <00:25:13.375>super <00:25:13.655>easy.

526
00:25:14.015 --> 00:25:16.785
<00:25:14.015>It's <00:25:14.248>the <00:25:14.481>expiry <00:25:14.715>attribute <00:25:14.948>with <00:25:15.181>API <00:25:15.415>keys.

527
00:25:16.805 --> 00:25:19.705
<00:25:16.805>You <00:25:16.945>can <00:25:17.085>somehow <00:25:17.225>store <00:25:17.365>it <00:25:17.505>in <00:25:17.645>the <00:25:17.785>database <00:25:17.925>with <00:25:18.065>a <00:25:18.205>grade

528
00:25:19.705 --> 00:25:22.905
<00:25:19.705>that <00:25:19.845>at <00:25:19.985>timestamp <00:25:20.125>with <00:25:20.264>when <00:25:20.405>it <00:25:20.544>should <00:25:20.685>be <00:25:20.824>not <00:25:20.965>valid <00:25:21.105>anymore

529
00:25:22.905 --> 00:25:23.905
<00:25:22.905>and <00:25:23.055>then <00:25:23.205>they <00:25:23.355>delete <00:25:23.504>it <00:25:23.655>from <00:25:23.805>there.

530
00:25:24.605 --> 00:25:26.945
<00:25:24.605>But <00:25:24.760>it's <00:25:24.916>on <00:25:25.071>custom <00:25:25.227>logic <00:25:25.382>you <00:25:25.538>have <00:25:25.693>to <00:25:25.849>do <00:25:26.005>yourself

531
00:25:29.025 --> 00:25:30.885
<00:25:29.025>and <00:25:29.305>from <00:25:29.585>utility <00:25:29.865>TLS <00:25:30.145>as <00:25:30.425>well.

532
00:25:33.215 --> 00:25:35.265
<00:25:33.215>Then <00:25:33.564>in <00:25:33.915>addition <00:25:34.264>to <00:25:34.615>authentication,

533
00:25:35.265 --> 00:25:36.705
<00:25:35.265>we <00:25:35.524>also <00:25:35.783>want <00:25:36.042>to <00:25:36.301>have <00:25:36.561>authorization.

534
00:25:37.695 --> 00:25:41.905
<00:25:37.695>That <00:25:37.850>means <00:25:38.006>a <00:25:38.161>customer <00:25:38.317>can <00:25:38.472>access <00:25:38.628>not <00:25:38.783>only <00:25:38.939>any <00:25:39.095>model,

535
00:25:41.965 --> 00:25:44.465
<00:25:41.965>but <00:25:42.120>only <00:25:42.276>a <00:25:42.431>very <00:25:42.587>specific <00:25:42.742>model <00:25:42.898>on <00:25:43.053>a <00:25:43.209>specific <00:25:43.365>route.

536
00:25:45.125 --> 00:25:48.465
<00:25:45.125>For <00:25:45.325>example, <00:25:45.525>some <00:25:45.725>customer <00:25:45.925>a <00:25:46.125>grades <00:25:46.325>a <00:25:46.525>token

537
00:25:48.465 --> 00:25:50.745
<00:25:48.465>and <00:25:48.639>he <00:25:48.814>wants <00:25:48.990>to <00:25:49.165>access <00:25:49.340>only <00:25:49.514>the <00:25:49.689>LAMA <00:25:49.865>model

538
00:25:50.745 --> 00:25:54.145
<00:25:50.745>because <00:25:50.945>he <00:25:51.145>thinks <00:25:51.344>the <00:25:51.544>model <00:25:51.745>is <00:25:51.945>too <00:25:52.145>expensive

539
00:25:54.245 --> 00:25:58.745
<00:25:54.245>or <00:25:54.385>maybe <00:25:54.524>not <00:25:54.665>good <00:25:54.804>enough <00:25:54.945>Before <00:25:55.084>he <00:25:55.225>gives <00:25:55.364>out <00:25:55.504>the, <00:25:55.645>uh,

540
00:25:58.745 --> 00:26:00.665
<00:25:58.745>authentication <00:25:59.094>token <00:25:59.445>to <00:25:59.794>its <00:26:00.145>developer.

541
00:26:00.685 --> 00:26:02.545
<00:26:00.685>So <00:26:00.860>he <00:26:01.034>want <00:26:01.210>to <00:26:01.385>have <00:26:01.560>some <00:26:01.735>restriction <00:26:01.909>into <00:26:02.085>it

542
00:26:02.935 --> 00:26:06.705
<00:26:02.935>with <00:26:03.090>various <00:26:03.246>tokens <00:26:03.401>we <00:26:03.557>can <00:26:03.712>encode <00:26:03.868>the <00:26:04.023>authorization <00:26:04.179>into <00:26:04.335>uh,

543
00:26:06.925 --> 00:26:09.145
<00:26:06.925>an <00:26:07.100>attribute <00:26:07.274>and <00:26:07.450>then <00:26:07.625>we <00:26:07.800>can <00:26:07.975>simply <00:26:08.149>check <00:26:08.325>it

544
00:26:11.315 --> 00:26:12.375
<00:26:11.315>in <00:26:11.792>API <00:26:12.269>keys.

545
00:26:12.435 --> 00:26:16.335
<00:26:12.435>We <00:26:12.668>need <00:26:12.901>a <00:26:13.135>database <00:26:13.368>access <00:26:13.601>for <00:26:13.835>it

546
00:26:16.355 --> 00:26:20.055
<00:26:16.355>and <00:26:16.530>mutual <00:26:16.705>analysis <00:26:16.880>supported, <00:26:17.055>but <00:26:17.230>I <00:26:17.405>guess <00:26:17.580>for <00:26:17.755>minutes.

547
00:26:21.715 --> 00:26:24.015
<00:26:21.715>So <00:26:21.822>at <00:26:21.930>the <00:26:22.038>end, <00:26:22.145>end <00:26:22.253>of <00:26:22.361>the <00:26:22.468>day <00:26:22.576>is <00:26:22.684>kind <00:26:22.791>of <00:26:22.899>a <00:26:23.007>mess <00:26:23.115>around.

548
00:26:25.395 --> 00:26:28.055
<00:26:25.395>We <00:26:25.550>are <00:26:25.706>using <00:26:25.861>barrier <00:26:26.017>tokens <00:26:26.172>because <00:26:26.328>we <00:26:26.483>really <00:26:26.639>like <00:26:26.795>it.

549
00:26:28.275 --> 00:26:29.335
<00:26:28.275>So <00:26:28.513>I'm <00:26:28.752>skipping <00:26:28.990>through <00:26:29.229>it

550
00:26:29.335 --> 00:26:30.735
<00:26:29.335>and <00:26:29.587>hopefully <00:26:29.839>this <00:26:30.091>is <00:26:30.342>really <00:26:30.594>good.

551
00:26:31.075 --> 00:26:33.695
<00:26:31.075>So <00:26:31.275>at <00:26:31.475>this <00:26:31.675>review <00:26:31.875>inference <00:26:32.075>engine, <00:26:32.275>we <00:26:32.475>evaluated

552
00:26:33.725 --> 00:26:35.615
<00:26:33.725>that <00:26:33.925>we <00:26:34.125>want <00:26:34.324>to <00:26:34.524>have <00:26:34.725>a <00:26:34.925>canner <00:26:35.125>redeployment.

553
00:26:35.875 --> 00:26:39.015
<00:26:35.875>We <00:26:36.015>want <00:26:36.155>to <00:26:36.295>have <00:26:36.435>our <00:26:36.575>auto <00:26:36.715>scaling <00:26:36.855>mechanism <00:26:36.995>based <00:26:37.135>on, <00:26:37.275>uh,

554
00:26:39.015 --> 00:26:43.565
<00:26:39.015>custom <00:26:39.190>metrics, <00:26:39.365>especially <00:26:39.540>on <00:26:39.715>the <00:26:39.890>Q <00:26:40.065>size <00:26:40.240>VLLM <00:26:40.415>have.

555
00:26:45.265 --> 00:26:47.405
<00:26:45.265>And <00:26:45.465>we <00:26:45.665>want <00:26:45.865>to <00:26:46.065>be <00:26:46.265>on <00:26:46.465>uh, <00:26:46.665>Kubernetes

556
00:26:47.625 --> 00:26:49.565
<00:26:47.625>and <00:26:47.858>there <00:26:48.091>are <00:26:48.325>multiple <00:26:48.558>solutions <00:26:48.791>to <00:26:49.025>that.

557
00:26:49.625 --> 00:26:51.125
<00:26:49.625>We <00:26:49.962>evaluated <00:26:50.300>three <00:26:50.637>different <00:26:50.975>things.

558
00:26:51.185 --> 00:26:53.645
<00:26:51.185>One <00:26:51.325>is <00:26:51.465>the <00:26:51.605>K <00:26:51.745>safe, <00:26:51.885>K <00:26:52.024>native, <00:26:52.165>and <00:26:52.304>the <00:26:52.445>Istio <00:26:52.585>thing.

559
00:26:54.225 --> 00:26:58.725
<00:26:54.225>It <00:26:54.380>fulfilled <00:26:54.536>all <00:26:54.691>of <00:26:54.847>our <00:26:55.002>needs, <00:26:55.158>which <00:26:55.313>I <00:26:55.469>already <00:26:55.625>mentioned.

560
00:26:58.725 --> 00:27:01.525
<00:26:58.725>Another <00:26:59.191>solution <00:26:59.658>is <00:27:00.125>Ray.

561
00:27:03.535 --> 00:27:05.875
<00:27:03.535>Um, <00:27:03.710>it <00:27:03.885>has <00:27:04.060>a <00:27:04.235>headnote <00:27:04.410>and <00:27:04.585>multiple <00:27:04.760>worker <00:27:04.935>noes.

562
00:27:05.945 --> 00:27:09.355
<00:27:05.945>It's <00:27:06.072>doing <00:27:06.199>a <00:27:06.326>quite <00:27:06.454>a <00:27:06.581>similar <00:27:06.708>thing <00:27:06.835>as <00:27:06.963>K <00:27:07.090>native <00:27:07.217>and <00:27:07.345>kerv.

563
00:27:10.255 --> 00:27:12.755
<00:27:10.255>And <00:27:10.430>finally, <00:27:10.605>you <00:27:10.780>maybe <00:27:10.955>don't <00:27:11.130>want <00:27:11.305>to <00:27:11.480>use <00:27:11.655>any

564
00:27:12.755 --> 00:27:14.235
<00:27:12.755>of <00:27:12.945>these <00:27:13.135>tools <00:27:13.325>at <00:27:13.516>all <00:27:13.706>because <00:27:13.896>they're <00:27:14.087>bringing

565
00:27:14.235 --> 00:27:15.355
<00:27:14.235>to <00:27:14.571>a <00:27:14.907>much <00:27:15.243>complexity.

566
00:27:15.355 --> 00:27:16.915
<00:27:15.355>So <00:27:15.555>you <00:27:15.755>can <00:27:15.955>simply <00:27:16.155>add <00:27:16.355>a <00:27:16.555>load <00:27:16.755>balancer

567
00:27:16.915 --> 00:27:17.915
<00:27:16.915>and <00:27:17.095>all <00:27:17.274>the <00:27:17.455>pos <00:27:17.635>you <00:27:17.815>have

568
00:27:18.295 --> 00:27:21.115
<00:27:18.295>and <00:27:18.435>Kubernetes <00:27:18.575>is <00:27:18.715>taking <00:27:18.855>care <00:27:18.995>about <00:27:19.135>the <00:27:19.275>load <00:27:19.415>balancing <00:27:19.555>in <00:27:19.695>a

569
00:27:21.115 --> 00:27:22.355
<00:27:21.115>round <00:27:21.673>robin <00:27:22.231>system.

570
00:27:22.655 --> 00:27:24.515
<00:27:22.655>But <00:27:22.888>this <00:27:23.121>has <00:27:23.355>downsides <00:27:23.588>that <00:27:23.821>you <00:27:24.055>can't

571
00:27:25.145 --> 00:27:28.915
<00:27:25.145>load <00:27:25.285>balancing <00:27:25.425>on <00:27:25.565>metrics <00:27:25.705>of <00:27:25.845>how <00:27:25.985>busy <00:27:26.125>the <00:27:26.264>LLM <00:27:26.405>model <00:27:26.545>is.

572
00:27:29.055 --> 00:27:31.235
<00:27:29.055>So <00:27:29.230>we <00:27:29.405>don't <00:27:29.580>want <00:27:29.755>to <00:27:29.930>redirect <00:27:30.105>every <00:27:30.280>single <00:27:30.455>request

573
00:27:31.975 --> 00:27:34.595
<00:27:31.975>to <00:27:32.130>a <00:27:32.286>busy <00:27:32.441>LLLM <00:27:32.597>model <00:27:32.752>in <00:27:32.908>this <00:27:33.063>round <00:27:33.219>robin <00:27:33.375>system,

574
00:27:35.595 --> 00:27:38.695
<00:27:35.595>but <00:27:35.828>rather <00:27:36.061>do <00:27:36.295>some, <00:27:36.528>uh, <00:27:36.761>smart <00:27:36.995>routing.

575
00:27:38.835 --> 00:27:40.215
<00:27:38.835>And <00:27:39.145>it <00:27:39.456>is <00:27:39.766>only <00:27:40.077>possible

576
00:27:40.445 --> 00:27:43.455
<00:27:40.445>with <00:27:40.645>solutions <00:27:40.845>like <00:27:41.044>K <00:27:41.245>native <00:27:41.445>and <00:27:41.645>case <00:27:41.845>of.

577
00:27:45.955 --> 00:27:48.055
<00:27:45.955>So <00:27:46.188>routing <00:27:46.421>is <00:27:46.655>already <00:27:46.888>the <00:27:47.121>last <00:27:47.355>point.

578
00:27:48.565 --> 00:27:50.215
<00:27:48.565>It's <00:27:48.845>a <00:27:49.125>very, <00:27:49.405>very <00:27:49.685>simple <00:27:49.965>component.

579
00:27:50.315 --> 00:27:52.095
<00:27:50.315>We <00:27:50.515>have <00:27:50.715>to <00:27:50.915>alize <00:27:51.115>the <00:27:51.315>re <00:27:51.515>request <00:27:51.715>body,

580
00:27:52.395 --> 00:27:55.175
<00:27:52.395>see <00:27:52.522>which <00:27:52.649>model <00:27:52.776>is <00:27:52.904>used, <00:27:53.031>then <00:27:53.158>have <00:27:53.285>an <00:27:53.413>internal <00:27:53.540>hash <00:27:53.667>map <00:27:53.795>of

581
00:27:55.385 --> 00:27:57.015
<00:27:55.385>where <00:27:55.735>should <00:27:56.085>the <00:27:56.435>request <00:27:56.785>go.

582
00:27:58.395 --> 00:28:01.455
<00:27:58.395>In <00:27:58.570>Kubernetes, <00:27:58.745>you <00:27:58.920>have <00:27:59.095>a <00:27:59.270>Kubernetes <00:27:59.445>service <00:27:59.620>in <00:27:59.795>lms,

583
00:28:01.455 --> 00:28:04.495
<00:28:01.455>you <00:28:01.582>maybe <00:28:01.709>have <00:28:01.836>some <00:28:01.964>IP <00:28:02.091>in, <00:28:02.218>uh, <00:28:02.345>the, <00:28:02.473>maybe <00:28:02.600>have <00:28:02.727>some <00:28:02.855>IP

584
00:28:05.435 --> 00:28:07.055
<00:28:05.435>stored <00:28:05.715>where <00:28:05.995>the <00:28:06.274>request <00:28:06.554>should <00:28:06.835>go.

585
00:28:07.435 --> 00:28:09.055
<00:28:07.435>So <00:28:07.715>this <00:28:07.995>is <00:28:08.274>very, <00:28:08.554>very <00:28:08.835>simple.

586
00:28:09.195 --> 00:28:12.615
<00:28:09.195>You <00:28:09.350>can <00:28:09.506>also <00:28:09.661>use <00:28:09.817>some <00:28:09.972>out <00:28:10.128>of <00:28:10.283>the <00:28:10.439>box <00:28:10.595>Component.

587
00:28:12.775 --> 00:28:14.295
<00:28:12.775>I <00:28:13.003>think <00:28:13.231>there's <00:28:13.459>a <00:28:13.687>project <00:28:13.915>called <00:28:14.143>LLMD,

588
00:28:14.295 --> 00:28:16.335
<00:28:14.295>which <00:28:14.528>now <00:28:14.761>has <00:28:14.995>this <00:28:15.228>model <00:28:15.461>router <00:28:15.695>order.

589
00:28:18.275 --> 00:28:21.055
<00:28:18.275>So, <00:28:18.415>and <00:28:18.555>this <00:28:18.695>brings <00:28:18.835>us <00:28:18.975>already, <00:28:19.115>uh, <00:28:19.255>to <00:28:19.395>the <00:28:19.535>last <00:28:19.675>points.

590
00:28:21.355 --> 00:28:23.975
<00:28:21.355>So <00:28:21.510>we <00:28:21.666>made <00:28:21.821>it <00:28:21.977>in <00:28:22.132>time, <00:28:22.288>which <00:28:22.443>is <00:28:22.599>really <00:28:22.755>great.

591
00:28:25.595 --> 00:28:29.955
<00:28:25.595>So <00:28:25.795>to <00:28:25.995>sum <00:28:26.195>it <00:28:26.395>up, <00:28:26.595>always <00:28:26.795>rate <00:28:26.995>limit

592
00:28:30.415 --> 00:28:31.555
<00:28:30.415>in <00:28:30.757>your <00:28:31.099>shared <00:28:31.441>system

593
00:28:31.555 --> 00:28:33.795
<00:28:31.555>because <00:28:31.755>a <00:28:31.955>single <00:28:32.155>customer <00:28:32.355>should <00:28:32.555>never <00:28:32.755>be <00:28:32.955>able

594
00:28:33.795 --> 00:28:35.355
<00:28:33.795>to <00:28:34.075>take <00:28:34.355>down <00:28:34.635>the <00:28:34.915>entire <00:28:35.195>system

595
00:28:35.975 --> 00:28:38.995
<00:28:35.975>and <00:28:36.115>make <00:28:36.254>it <00:28:36.395>unusable <00:28:36.534>for <00:28:36.675>all <00:28:36.814>the <00:28:36.955>other <00:28:37.094>customers <00:28:37.235>you <00:28:37.375>have.

596
00:28:39.545 --> 00:28:41.515
<00:28:39.545>This <00:28:39.778>applies <00:28:40.011>generally <00:28:40.245>to <00:28:40.478>any <00:28:40.711>kind <00:28:40.945>of

597
00:28:42.825 --> 00:28:44.075
<00:28:42.825>SaaS <00:28:43.106>solution <00:28:43.387>you <00:28:43.668>are <00:28:43.950>building.

598
00:28:45.335 --> 00:28:50.325
<00:28:45.335>Um, <00:28:46.735>restrict

599
00:28:50.325 --> 00:28:52.205
<00:28:50.325>the <00:28:50.558>alpha <00:28:50.791>tokens <00:28:51.025>and <00:28:51.258>LLM <00:28:51.491>is <00:28:51.725>producing.

600
00:28:52.235 --> 00:28:53.445
<00:28:52.235>This <00:28:52.507>goes <00:28:52.779>into <00:28:53.051>the <00:28:53.323>direction

601
00:28:53.445 --> 00:28:56.525
<00:28:53.445>of <00:28:53.600>having <00:28:53.756>super <00:28:53.911>large <00:28:54.067>request <00:28:54.222>bodies <00:28:54.378>in <00:28:54.533>a <00:28:54.689>synchronous <00:28:54.845>rest

602
00:28:56.645 --> 00:28:59.365
<00:28:56.645>API <00:28:56.820>the <00:28:56.995>first <00:28:57.170>thing <00:28:57.345>you're <00:28:57.520>doing <00:28:57.695>is <00:28:57.870>introducing <00:28:58.045>paging.

603
00:28:59.985 --> 00:29:03.245
<00:28:59.985>In <00:29:00.125>an <00:29:00.264>LLM <00:29:00.405>context, <00:29:00.544>you <00:29:00.685>are <00:29:00.824>limiting <00:29:00.965>the <00:29:01.104>AL <00:29:01.245>output <00:29:01.385>tokens.

604
00:29:03.305 --> 00:29:05.645
<00:29:03.305>So <00:29:03.445>if <00:29:03.585>a <00:29:03.725>user <00:29:03.865>is <00:29:04.005>saying, <00:29:04.145>please <00:29:04.285>generate <00:29:04.425>me <00:29:04.565>a <00:29:04.705>book,

605
00:29:06.425 --> 00:29:08.125
<00:29:06.425>it <00:29:06.705>doesn't <00:29:06.985>generate <00:29:07.264>the <00:29:07.544>entire <00:29:07.825>book.

606
00:29:08.145 --> 00:29:11.525
<00:29:08.145>So <00:29:08.320>having <00:29:08.495>the <00:29:08.670>super <00:29:08.845>huge <00:29:09.020>request, <00:29:09.195>uh, <00:29:09.370>response <00:29:09.545>body,

607
00:29:12.625 --> 00:29:14.725
<00:29:12.625>but <00:29:12.975>only <00:29:13.325>like <00:29:13.675>4,000 <00:29:14.025>tokens.

608
00:29:14.905 --> 00:29:16.525
<00:29:14.905>So <00:29:15.185>one <00:29:15.465>paragraph <00:29:15.745>in <00:29:16.024>the <00:29:16.305>book

609
00:29:16.585 --> 00:29:18.325
<00:29:16.585>and <00:29:16.785>then <00:29:16.985>a <00:29:17.185>user <00:29:17.385>need <00:29:17.585>to <00:29:17.785>request <00:29:17.985>again,

610
00:29:18.435 --> 00:29:20.685
<00:29:18.435>give <00:29:18.610>me <00:29:18.784>the <00:29:18.960>next <00:29:19.135>paragraph, <00:29:19.310>gimme <00:29:19.485>the <00:29:19.659>next <00:29:19.835>paragraph

611
00:29:21.795 --> 00:29:23.885
<00:29:21.795>because <00:29:22.028>we <00:29:22.261>want <00:29:22.495>want <00:29:22.728>to <00:29:22.961>avoid <00:29:23.195>that.

612
00:29:23.885 --> 00:29:25.205
<00:29:23.885>The <00:29:24.083>model <00:29:24.281>is <00:29:24.479>busy <00:29:24.677>all <00:29:24.875>the <00:29:25.072>time

613
00:29:25.395 --> 00:29:26.925
<00:29:25.395>that <00:29:25.739>this <00:29:26.083>is, <00:29:26.427>uh, <00:29:26.772>optimization.

614
00:29:26.925 --> 00:29:29.445
<00:29:26.925>We <00:29:27.158>do <00:29:27.391>in, <00:29:27.625>uh, <00:29:27.858>synchronous <00:29:28.091>stress <00:29:28.325>APIs

615
00:29:29.445 --> 00:29:30.565
<00:29:29.445>all <00:29:29.697>the <00:29:29.949>time <00:29:30.201>with <00:29:30.453>paging.

616
00:29:30.565 --> 00:29:33.845
<00:29:30.565>And <00:29:30.740>then <00:29:30.915>LM <00:29:31.090>conduct <00:29:31.265>is <00:29:31.440>called <00:29:31.615>a <00:29:31.790>slightly <00:29:31.965>different.

617
00:29:35.305 --> 00:29:37.885
<00:29:35.305>You <00:29:35.460>can <00:29:35.616>fix <00:29:35.771>what <00:29:35.927>you <00:29:36.082>can <00:29:36.238>see, <00:29:36.393>always <00:29:36.549>have <00:29:36.705>observability,

618
00:29:38.035 --> 00:29:39.805
<00:29:38.035>have <00:29:38.315>all <00:29:38.595>the <00:29:38.875>metrics, <00:29:39.155>logs, <00:29:39.435>traces.

619
00:29:40.385 --> 00:29:43.845
<00:29:40.385>If <00:29:40.501>you <00:29:40.618>have <00:29:40.735>K <00:29:40.851>kerv <00:29:40.968>you're <00:29:41.085>getting <00:29:41.201>is <00:29:41.318>still <00:29:41.435>out <00:29:41.551>of <00:29:41.668>the <00:29:41.785>box.

620
00:29:43.985 --> 00:29:45.725
<00:29:43.985>It <00:29:44.264>produces <00:29:44.544>a <00:29:44.824>bunch <00:29:45.104>of <00:29:45.385>logs

621
00:29:46.145 --> 00:29:49.525
<00:29:46.145>and <00:29:46.320>traces <00:29:46.495>so <00:29:46.670>are <00:29:46.845>never <00:29:47.020>aligned <00:29:47.195>during <00:29:47.370>any <00:29:47.545>accident.

622
00:29:50.305 --> 00:29:51.645
<00:29:50.305>Um, <00:29:50.606>the <00:29:50.908>sign <00:29:51.209>with <00:29:51.511>modularity.

623
00:29:51.785 --> 00:29:53.605
<00:29:51.785>So <00:29:51.925>we <00:29:52.065>want <00:29:52.205>to <00:29:52.345>stick <00:29:52.485>in <00:29:52.625>and <00:29:52.765>take <00:29:52.905>out <00:29:53.045>every <00:29:53.185>single

624
00:29:53.605 --> 00:29:54.685
<00:29:53.605>component <00:29:53.929>for <00:29:54.253>the <00:29:54.577>wheel.

625
00:29:54.885 --> 00:29:58.045
<00:29:54.885>M <00:29:55.012>you <00:29:55.139>maybe <00:29:55.266>want <00:29:55.394>to <00:29:55.521>do <00:29:55.648>this, <00:29:55.775>um, <00:29:55.903>all <00:29:56.030>Lama <00:29:56.157>for <00:29:56.285>example.

626
00:29:59.225 --> 00:30:00.445
<00:29:59.225>So <00:29:59.591>be <00:29:59.956>really <00:30:00.322>flexible.

627
00:30:00.445 --> 00:30:03.245
<00:30:00.445>Yeah, <00:30:00.600>that <00:30:00.756>m <00:30:00.911>landscape <00:30:01.067>is <00:30:01.222>like <00:30:01.378>evolving <00:30:01.533>really, <00:30:01.689>really <00:30:01.845>fast.

628
00:30:03.335 --> 00:30:06.485
<00:30:03.335>There <00:30:03.535>are <00:30:03.735>new <00:30:03.935>tools <00:30:04.135>occurring <00:30:04.335>every <00:30:04.535>single <00:30:04.735>day.

629
00:30:08.595 --> 00:30:09.895
<00:30:08.595>Do <00:30:08.887>benchmarks <00:30:09.180>all <00:30:09.472>the <00:30:09.764>time

630
00:30:09.955 --> 00:30:11.655
<00:30:09.955>before <00:30:10.235>you <00:30:10.514>are <00:30:10.794>introduce <00:30:11.074>something <00:30:11.355>new.

631
00:30:11.875 --> 00:30:14.375
<00:30:11.875>Always <00:30:12.225>run <00:30:12.575>your <00:30:12.925>performance <00:30:13.275>test

632
00:30:16.845 --> 00:30:19.715
<00:30:16.845>since <00:30:17.020>it's <00:30:17.195>really <00:30:17.370>expensive <00:30:17.545>to <00:30:17.720>have <00:30:17.895>a <00:30:18.070>testing <00:30:18.245>environment

633
00:30:19.865 --> 00:30:21.435
<00:30:19.865>with <00:30:20.098>all <00:30:20.331>the <00:30:20.565>models <00:30:20.798>you <00:30:21.031>are <00:30:21.265>providing.

634
00:30:21.435 --> 00:30:22.555
<00:30:21.435>This <00:30:21.771>is <00:30:22.107>almost <00:30:22.443>impossible.

635
00:30:22.935 --> 00:30:24.995
<00:30:22.935>So <00:30:23.215>we <00:30:23.495>are <00:30:23.774>testing <00:30:24.054>in <00:30:24.335>production

636
00:30:24.995 --> 00:30:26.315
<00:30:24.995>with <00:30:25.390>this <00:30:25.787>canary <00:30:26.183>deployments.

637
00:30:26.325 --> 00:30:29.275
<00:30:26.325>Every <00:30:26.465>time <00:30:26.605>we're <00:30:26.745>rolling <00:30:26.885>up <00:30:27.025>to <00:30:27.165>a <00:30:27.305>new <00:30:27.445>version, <00:30:27.585>we <00:30:27.725>are

638
00:30:29.955 --> 00:30:31.395
<00:30:29.955>spawning <00:30:30.279>up <00:30:30.603>a <00:30:30.927>new <00:30:31.251>replica,

639
00:30:31.725 --> 00:30:34.075
<00:30:31.725>doing <00:30:31.899>our <00:30:32.074>intent <00:30:32.250>end-to-end <00:30:32.425>test <00:30:32.600>on <00:30:32.774>this <00:30:32.949>new <00:30:33.125>replica,

640
00:30:34.075 --> 00:30:36.515
<00:30:34.075>then <00:30:34.230>routing <00:30:34.386>10% <00:30:34.541>of <00:30:34.697>the <00:30:34.852>traffic, <00:30:35.008>20% <00:30:35.163>of <00:30:35.319>the <00:30:35.475>traffic,

641
00:30:36.535 --> 00:30:37.955
<00:30:36.535>and <00:30:36.748>so <00:30:36.961>on <00:30:37.174>to <00:30:37.387>this <00:30:37.600>new <00:30:37.812>version.

642
00:30:39.345 --> 00:30:41.285
<00:30:39.345>And <00:30:39.545>in <00:30:39.745>that <00:30:39.945>way <00:30:40.145>you're <00:30:40.345>saving <00:30:40.545>a <00:30:40.745>lot

643
00:30:41.285 --> 00:30:43.685
<00:30:41.285>of <00:30:41.485>money <00:30:41.685>in <00:30:41.885>your <00:30:42.085>development <00:30:42.285>and <00:30:42.485>saging <00:30:42.685>environment

644
00:30:43.685 --> 00:30:46.885
<00:30:43.685>because <00:30:43.840>there's <00:30:43.996>absolutely <00:30:44.151>no <00:30:44.307>way <00:30:44.462>you <00:30:44.618>can <00:30:44.773>afford <00:30:44.929>hosting <00:30:45.085>all

645
00:30:46.885 --> 00:30:50.205
<00:30:46.885>the <00:30:47.165>models <00:30:47.445>in, <00:30:47.725>uh, <00:30:48.004>this <00:30:48.285>environment.

646
00:30:50.205 --> 00:30:52.205
<00:30:50.205>But <00:30:50.438>you <00:30:50.671>still <00:30:50.905>want <00:30:51.138>to <00:30:51.371>have <00:30:51.605>insurance

647
00:30:52.205 --> 00:30:54.045
<00:30:52.205>that <00:30:52.438>is <00:30:52.671>actually <00:30:52.905>working <00:30:53.138>as <00:30:53.371>you <00:30:53.605>think.

648
00:30:54.045 --> 00:30:55.885
<00:30:54.045>You <00:30:54.395>think <00:30:54.745>they <00:30:55.095>are <00:30:55.445>working.

649
00:30:56.265 --> 00:30:57.885
<00:30:56.265>And <00:30:56.440>I <00:30:56.615>think <00:30:56.790>can <00:30:56.965>is <00:30:57.140>the <00:30:57.315>way <00:30:57.490>to <00:30:57.665>go.

650
00:31:00.275 --> 00:31:03.055
<00:31:00.275>And <00:31:00.430>I <00:31:00.586>said <00:31:00.741>somewhere <00:31:00.897>in <00:31:01.052>the <00:31:01.208>middle <00:31:01.363>of <00:31:01.519>public <00:31:01.675>benchmarks,

651
00:31:03.375 --> 00:31:08.135
<00:31:03.375>UNE <00:31:03.550>equals <00:31:03.725>three <00:31:03.900>life <00:31:04.075>always <00:31:04.250>do <00:31:04.425>your <00:31:04.600>own <00:31:04.775>benchmarks.

652
00:31:08.285 --> 00:31:09.735
<00:31:08.285>They <00:31:08.611>are <00:31:08.937>usually <00:31:09.263>quite <00:31:09.590>different

653
00:31:09.765 --> 00:31:10.895
<00:31:09.765>what <00:31:09.968>you <00:31:10.171>see <00:31:10.375>in <00:31:10.578>the <00:31:10.782>internet

654
00:31:11.435 --> 00:31:14.335
<00:31:11.435>and <00:31:11.635>you <00:31:11.835>are, <00:31:12.034>you <00:31:12.235>need <00:31:12.435>to <00:31:12.635>make <00:31:12.835>sure

655
00:31:14.805 --> 00:31:17.695
<00:31:14.805>that <00:31:14.960>the <00:31:15.116>LM <00:31:15.271>behaves <00:31:15.427>like <00:31:15.582>the <00:31:15.738>customer <00:31:15.893>think <00:31:16.049>it <00:31:16.205>behaves.

656
00:31:17.695 --> 00:31:21.575
<00:31:17.695>And <00:31:17.835>you <00:31:17.975>can <00:31:18.115>only <00:31:18.254>ensure <00:31:18.395>that <00:31:18.534>by <00:31:18.675>doing <00:31:18.814>your <00:31:18.955>own <00:31:19.095>benchmarks.