WEBVTT

1
00:00:03.380 --> 00:00:07.200
<00:00:03.380>Hi, <00:00:03.496>and <00:00:03.613>thanks <00:00:03.730>for <00:00:03.846>watching. <00:00:03.963>I'm <00:00:04.080>Torsten <00:00:04.196>Rieß, <00:00:04.313>and <00:00:04.430>this <00:00:04.546>is <00:00:04.663>a <00:00:04.779>talk

2
00:00:07.280 --> 00:00:10.720
<00:00:07.280>about <00:00:07.480>keeping <00:00:07.680>fleet <00:00:07.880>management <00:00:08.080>and <00:00:08.280>telematics <00:00:08.480>platform <00:00:08.680>simple.

3
00:00:11.140 --> 00:00:14.580
<00:00:11.140>A <00:00:11.267>tour <00:00:11.394>of <00:00:11.521>how <00:00:11.649>we <00:00:11.776>actually <00:00:11.903>run <00:00:12.030>our <00:00:12.158>stack <00:00:12.285>here <00:00:12.412>at <00:00:12.540>Trackonix,

4
00:00:15.080 --> 00:00:17.500
<00:00:15.080>told <00:00:15.280>as <00:00:15.480>stories <00:00:15.680>rather <00:00:15.880>than <00:00:16.079>a <00:00:16.280>product <00:00:16.480>pitch.

5
00:00:18.580 --> 00:00:22.440
<00:00:18.580>The <00:00:18.735>thread <00:00:18.891>running <00:00:19.046>through <00:00:19.202>almost <00:00:19.357>every <00:00:19.513>decision <00:00:19.668>is <00:00:19.824>one <00:00:19.979>question:

6
00:00:23.140 --> 00:00:25.500
<00:00:23.140>Can <00:00:23.373>we <00:00:23.606>avoid <00:00:23.840>adding <00:00:24.073>another <00:00:24.306>moving <00:00:24.540>part?

7
00:00:26.440 --> 00:00:30.180
<00:00:26.440>I'll <00:00:26.580>start <00:00:26.720>with <00:00:26.860>the <00:00:27.000>philosophy <00:00:27.140>and <00:00:27.280>the <00:00:27.420>infrastructure, <00:00:27.560>and <00:00:27.700>then <00:00:27.840>walk

8
00:00:30.240 --> 00:00:33.720
<00:00:30.240>through <00:00:30.415>four <00:00:30.590>short <00:00:30.764>deep <00:00:30.939>dives, <00:00:31.115>a <00:00:31.290>Kalman <00:00:31.465>filter, <00:00:31.639>trip

9
00:00:33.780 --> 00:00:37.720
<00:00:33.780>detection, <00:00:33.980>time-variant <00:00:34.180>graphs <00:00:34.380>for <00:00:34.580>trailers, <00:00:34.780>and <00:00:34.980>cold <00:00:35.180>storage

10
00:00:37.920 --> 00:00:41.280
<00:00:37.920>you <00:00:38.075>query <00:00:38.231>from <00:00:38.386>the <00:00:38.542>browser, <00:00:38.697>and <00:00:38.853>then <00:00:39.008>zoom <00:00:39.164>back <00:00:39.320>out.

11
00:00:42.100 --> 00:00:45.980
<00:00:42.100>If <00:00:42.240>the <00:00:42.380>sequel <00:00:42.520>goes <00:00:42.660>by <00:00:42.800>quickly, <00:00:42.940>the <00:00:43.080>headlines <00:00:43.220>on <00:00:43.360>each <00:00:43.500>slide

12
00:00:46.020 --> 00:00:47.260
<00:00:46.020>is <00:00:46.243>the <00:00:46.466>part <00:00:46.689>to <00:00:46.912>hold <00:00:47.135>onto.

13
00:00:48.900 --> 00:00:52.500
<00:00:48.900>Quick <00:00:49.040>word <00:00:49.180>about <00:00:49.320>me. <00:00:49.460>I'm <00:00:49.600>a <00:00:49.739>mathematician <00:00:49.879>by <00:00:50.019>training, <00:00:50.160>but <00:00:50.300>my

14
00:00:52.540 --> 00:00:56.420
<00:00:52.540>day-to-day <00:00:52.714>work <00:00:52.890>is <00:00:53.065>across <00:00:53.240>dev <00:00:53.415>sec <00:00:53.589>ops, <00:00:53.765>system <00:00:53.940>architecture,

15
00:00:56.620 --> 00:01:00.379
<00:00:56.620>security. <00:00:56.820>I <00:00:57.019>teach <00:00:57.220>geo-information <00:00:57.419>systems <00:00:57.620>at <00:00:57.820>a <00:00:58.019>college

16
00:01:00.600 --> 00:01:01.340
<00:01:00.600>in <00:01:01.266>Konstanz.

17
00:01:03.880 --> 00:01:07.600
<00:01:03.880>I <00:01:04.019>mentioned <00:01:04.160>the <00:01:04.300>mathematics <00:01:04.440>because <00:01:04.580>that's <00:01:04.720>why <00:01:04.860>I'm <00:01:05.000>comfortable <00:01:05.140>putting <00:01:05.280>something

18
00:01:07.640 --> 00:01:11.480
<00:01:07.640>like <00:01:07.756>a <00:01:07.873>Kalman <00:01:07.990>filter <00:01:08.106>or <00:01:08.223>a <00:01:08.340>bit <00:01:08.456>of <00:01:08.573>graph <00:01:08.690>theory <00:01:08.806>directly <00:01:08.923>into <00:01:09.040>the

19
00:01:11.520 --> 00:01:12.040
database,

20
00:01:12.940 --> 00:01:15.660
<00:01:12.940>and <00:01:13.115>the <00:01:13.289>breadth <00:01:13.465>is <00:01:13.640>really <00:01:13.815>the <00:01:13.990>thesis <00:01:14.164>in <00:01:14.340>miniature.

21
00:01:16.500 --> 00:01:20.400
<00:01:16.500>On <00:01:16.655>a <00:01:16.811>small <00:01:16.966>team, <00:01:17.122>one <00:01:17.277>person <00:01:17.433>wears <00:01:17.588>all <00:01:17.744>those <00:01:17.900>hats,

22
00:01:20.680 --> 00:01:22.820
<00:01:20.680>so <00:01:21.030>simplicity <00:01:21.380>isn't <00:01:21.730>the <00:01:22.080>preference.

23
00:01:23.460 --> 00:01:27.070
<00:01:23.460>Every <00:01:23.615>moving <00:01:23.771>part <00:01:23.926>is <00:01:24.082>something <00:01:24.237>I <00:01:24.393>personally <00:01:24.548>have <00:01:24.704>to <00:01:24.860>operate,

24
00:01:27.320 --> 00:01:29.360
<00:01:27.320>secure, <00:01:28.020>and <00:01:28.720>debug.

25
00:01:31.080 --> 00:01:34.660
<00:01:31.080>Who <00:01:31.255>we <00:01:31.429>are? <00:01:31.605>We <00:01:31.780>are <00:01:31.955>Trackonix, <00:01:32.130>a <00:01:32.304>Swiss <00:01:32.480>software-as-a-service

26
00:01:34.760 --> 00:01:37.500
<00:01:34.759>company <00:01:35.039>doing <00:01:35.320>fleet <00:01:35.600>management <00:01:35.880>and <00:01:36.160>telematics.

27
00:01:37.720 --> 00:01:41.260
<00:01:37.720>We <00:01:37.895>support <00:01:38.070>a <00:01:38.245>wide <00:01:38.420>variety <00:01:38.595>of <00:01:38.770>vehicles <00:01:38.945>from <00:01:39.120>light

28
00:01:41.300 --> 00:01:44.800
<00:01:41.300>EVs <00:01:41.475>and <00:01:41.649>bikes <00:01:41.825>to <00:01:42.000>trucks, <00:01:42.175>trailers, <00:01:42.350>winter <00:01:42.524>service <00:01:42.700>vehicles,

29
00:01:45.460 --> 00:01:49.000
<00:01:45.460>and <00:01:45.660>even <00:01:45.860>racing <00:01:46.060>cars. <00:01:46.260>So <00:01:46.460>the <00:01:46.660>data <00:01:46.860>is

30
00:01:49.080 --> 00:01:50.680
<00:01:49.080>messy <00:01:49.780>and <00:01:50.480>heterogeneous.

31
00:01:51.880 --> 00:01:55.700
<00:01:51.880>Around <00:01:52.054>14,000 <00:01:52.229>vehicles <00:01:52.405>are <00:01:52.580>live <00:01:52.755>today, <00:01:52.929>and <00:01:53.104>at <00:01:53.280>the

32
00:01:55.760 --> 00:01:59.580
<00:01:55.759>daily <00:01:55.960>peak, <00:01:56.160>we <00:01:56.359>ingest <00:01:56.559>roughly <00:01:56.759>150 <00:01:56.960>records <00:01:57.160>per

33
00:01:59.680 --> 00:02:00.020
second.

34
00:02:02.700 --> 00:02:06.440
<00:02:02.700>Being <00:02:02.855>honest <00:02:03.011>about <00:02:03.166>that, <00:02:03.322>it's <00:02:03.477>a <00:02:03.633>real <00:02:03.788>production <00:02:03.944>load, <00:02:04.100>but

35
00:02:06.500 --> 00:02:10.301
<00:02:06.500>it <00:02:06.675>isn't <00:02:06.850>web <00:02:07.025>scale. <00:02:07.200>A <00:02:07.375>single <00:02:07.550>well-tuned <00:02:07.725>Postgres <00:02:07.900>can

36
00:02:10.361 --> 00:02:11.800
<00:02:10.360>handle <00:02:11.008>that <00:02:11.656>comfortably.

37
00:02:13.780 --> 00:02:17.580
<00:02:13.780>But <00:02:13.935>we're <00:02:14.091>a <00:02:14.246>small <00:02:14.402>team, <00:02:14.557>and <00:02:14.713>now <00:02:14.868>our <00:02:15.024>scarcest <00:02:15.180>resource

38
00:02:17.640 --> 00:02:20.180
<00:02:17.640>isn't <00:02:17.920>CPU <00:02:18.200>or <00:02:18.480>storage, <00:02:18.760>it's <00:02:19.040>attention.

39
00:02:21.010 --> 00:02:24.820
<00:02:21.010>So <00:02:21.149>our <00:02:21.290>guiding <00:02:21.429>principle <00:02:21.570>is <00:02:21.709>to <00:02:21.850>minimize <00:02:21.989>the <00:02:22.130>number <00:02:22.269>of <00:02:22.410>independent

40
00:02:24.880 --> 00:02:28.080
<00:02:24.880>things <00:02:25.079>that <00:02:25.280>can <00:02:25.480>break. <00:02:25.680>It's <00:02:25.880>a <00:02:26.079>deliberate <00:02:26.280>trade-off,

41
00:02:28.720 --> 00:02:32.020
<00:02:28.720>and <00:02:28.875>I'll <00:02:29.031>be <00:02:29.186>honest <00:02:29.342>later <00:02:29.497>about <00:02:29.653>where <00:02:29.808>the <00:02:29.964>cost <00:02:30.120>is.

42
00:02:35.160 --> 00:02:37.680
<00:02:35.160>Here's <00:02:35.440>two <00:02:35.720>real <00:02:36.000>screenshots, <00:02:36.280>not <00:02:36.560>mock-ups.

43
00:02:38.460 --> 00:02:40.620
<00:02:38.460>The <00:02:38.635>one <00:02:38.810>is <00:02:38.985>the <00:02:39.160>live <00:02:39.335>fleet <00:02:39.510>on <00:02:39.685>our <00:02:39.860>map.

44
00:02:41.080 --> 00:02:44.620
<00:02:41.079>The <00:02:41.235>bottom <00:02:41.391>is <00:02:41.546>our <00:02:41.702>actually <00:02:41.857>monitoring <00:02:42.013>graph <00:02:42.168>of <00:02:42.324>records <00:02:42.480>per

45
00:02:44.720 --> 00:02:48.600
<00:02:44.720>second <00:02:44.859>over <00:02:45.000>a <00:02:45.140>single <00:02:45.280>day. <00:02:45.420>Flat <00:02:45.560>overnight, <00:02:45.700>climbing <00:02:45.840>through <00:02:45.980>the <00:02:46.120>working

46
00:02:48.660 --> 00:02:52.120
<00:02:48.660>day <00:02:48.859>with <00:02:49.060>the <00:02:49.260>daytime <00:02:49.460>band <00:02:49.660>sitting <00:02:49.859>around <00:02:50.060>150.

47
00:02:53.620 --> 00:02:57.560
<00:02:53.620>That <00:02:53.795>long <00:02:53.970>spike <00:02:54.145>is <00:02:54.320>near <00:02:54.495>400 <00:02:54.670>per <00:02:54.845>second, <00:02:55.020>is

48
00:02:57.620 --> 00:03:00.880
<00:02:57.620>a <00:02:57.795>firmware <00:02:57.970>upgrade <00:02:58.145>window <00:02:58.320>across <00:02:58.495>many <00:02:58.670>devices <00:02:58.845>at <00:02:59.020>once.

49
00:03:01.540 --> 00:03:04.790
<00:03:01.540>No <00:03:01.739>data <00:03:01.940>ingest <00:03:02.140>but <00:03:02.340>load <00:03:02.540>all <00:03:02.739>the <00:03:02.940>same.

50
00:03:05.720 --> 00:03:08.900
<00:03:05.720>Apart <00:03:05.875>from <00:03:06.031>that, <00:03:06.186>ingest <00:03:06.342>just <00:03:06.497>follows <00:03:06.653>the <00:03:06.808>daily <00:03:06.964>working <00:03:07.120>day.

51
00:03:10.300 --> 00:03:14.180
<00:03:10.300>The <00:03:10.407>point <00:03:10.515>to <00:03:10.623>hold <00:03:10.730>on <00:03:10.838>to <00:03:10.946>this <00:03:11.053>is <00:03:11.161>the <00:03:11.269>entire <00:03:11.376>write <00:03:11.484>load <00:03:11.592>the <00:03:11.700>system

52
00:03:14.260 --> 00:03:18.160
<00:03:14.260>has <00:03:14.399>to <00:03:14.540>absorb, <00:03:14.679>and <00:03:14.820>it's <00:03:14.959>what <00:03:15.100>every <00:03:15.239>in <00:03:15.380>the <00:03:15.519>database <00:03:15.660>decision

53
00:03:18.900 --> 00:03:20.320
<00:03:18.900>later <00:03:19.326>is <00:03:19.752>sized <00:03:20.178>against.

54
00:03:23.520 --> 00:03:27.200
<00:03:23.520>If <00:03:23.695>you're <00:03:23.870>asked <00:03:24.045>about <00:03:24.220>the <00:03:24.395>architecture <00:03:24.570>of <00:03:24.745>a <00:03:24.920>telematic

55
00:03:27.220 --> 00:03:27.640
system,

56
00:03:28.820 --> 00:03:32.680
<00:03:28.820>something <00:03:28.920>like <00:03:29.019>this <00:03:29.120>is <00:03:29.220>probably <00:03:29.320>what <00:03:29.420>you <00:03:29.519>would <00:03:29.620>sketch <00:03:29.720>on <00:03:29.820>a <00:03:29.920>whiteboard <00:03:30.019>for <00:03:30.120>a <00:03:30.220>system

57
00:03:32.700 --> 00:03:35.760
<00:03:32.700>like <00:03:32.933>this, <00:03:33.166>and <00:03:33.399>every <00:03:33.633>box <00:03:33.866>is <00:03:34.100>on

58
00:03:35.900 --> 00:03:38.660
<00:03:35.900>its <00:03:36.366>own <00:03:36.833>individually <00:03:37.300>reasonable.

59
00:03:39.440 --> 00:03:43.380
<00:03:39.440>The <00:03:39.640>trouble <00:03:39.840>is <00:03:40.040>emergent. <00:03:40.240>It's <00:03:40.440>not <00:03:40.640>the <00:03:40.840>boxes,

60
00:03:43.740 --> 00:03:45.700
<00:03:43.740>it's <00:03:44.090>the <00:03:44.440>errors <00:03:44.790>between <00:03:45.140>them.

61
00:03:46.660 --> 00:03:50.540
<00:03:46.660>Each <00:03:46.787>integration <00:03:46.914>point <00:03:47.041>is <00:03:47.169>a <00:03:47.296>place <00:03:47.423>where <00:03:47.550>data <00:03:47.678>can <00:03:47.805>be <00:03:47.932>lost, <00:03:48.060>duplicated,

62
00:03:50.720 --> 00:03:54.700
<00:03:50.720>or <00:03:50.859>drift <00:03:51.000>out <00:03:51.140>of <00:03:51.280>sync. <00:03:51.420>The <00:03:51.560>cost <00:03:51.700>nobody <00:03:51.840>budget <00:03:51.980>for <00:03:52.120>is

63
00:03:54.760 --> 00:03:56.980
<00:03:54.760>keeping <00:03:55.109>all <00:03:55.459>these <00:03:55.810>stores <00:03:56.160>consistent.

64
00:03:57.560 --> 00:04:00.900
<00:03:57.560>The <00:03:57.687>moment <00:03:57.814>the <00:03:57.941>same <00:03:58.069>fact <00:03:58.196>lives <00:03:58.323>in <00:03:58.450>two <00:03:58.578>places, <00:03:58.705>you <00:03:58.832>own <00:03:58.960>a

65
00:04:00.960 --> 00:04:03.080
<00:04:00.960>reconciliation <00:04:01.660>problem <00:04:02.360>forever.

66
00:04:03.980 --> 00:04:07.000
<00:04:03.980>So <00:04:04.179>the <00:04:04.380>natural <00:04:04.579>question <00:04:04.780>is, <00:04:04.980>what's <00:04:05.179>the <00:04:05.380>alternative?

67
00:04:09.840 --> 00:04:13.220
<00:04:09.840>Our <00:04:10.015>answer <00:04:10.190>rests <00:04:10.365>on <00:04:10.540>two <00:04:10.715>old <00:04:10.890>ideas. <00:04:11.065>The <00:04:11.240>Unix

68
00:04:13.240 --> 00:04:13.880
philosophy,

69
00:04:14.980 --> 00:04:18.860
<00:04:14.980>one <00:04:15.119>tool, <00:04:15.260>one <00:04:15.399>job <00:04:15.540>composed <00:04:15.679>together <00:04:15.820>and <00:04:15.959>a <00:04:16.099>single <00:04:16.240>source <00:04:16.380>of

70
00:04:18.899 --> 00:04:22.320
<00:04:18.899>truth. <00:04:19.099>One <00:04:19.298>place <00:04:19.499>where <00:04:19.699>a <00:04:19.899>given <00:04:20.099>fact <00:04:20.298>lives.

71
00:04:22.400 --> 00:04:26.220
<00:04:22.400>Neither <00:04:22.599>is <00:04:22.799>novel. <00:04:23.000>The <00:04:23.200>discipline <00:04:23.400>holding <00:04:23.599>the <00:04:23.799>line

72
00:04:26.280 --> 00:04:28.560
<00:04:26.280>when <00:04:26.513>it's <00:04:26.746>tempting <00:04:26.979>to <00:04:27.213>add <00:04:27.446>a <00:04:27.679>box.

73
00:04:31.040 --> 00:04:35.020
<00:04:31.040>And <00:04:31.147>you <00:04:31.255>may <00:04:31.363>already <00:04:31.470>feel <00:04:31.578>the <00:04:31.686>tension <00:04:31.793>here <00:04:31.901>in <00:04:32.009>one <00:04:32.116>tool, <00:04:32.224>one <00:04:32.332>job <00:04:32.440>versus,

74
00:04:36.320 --> 00:04:40.080
<00:04:36.320>as <00:04:36.475>I'll <00:04:36.631>seem <00:04:36.786>to <00:04:36.942>argue, <00:04:37.097>using <00:04:37.253>Postgres <00:04:37.408>for <00:04:37.564>almost <00:04:37.719>everything.

75
00:04:40.680 --> 00:04:43.130
<00:04:40.680>I <00:04:40.835>want <00:04:40.991>to <00:04:41.146>be <00:04:41.302>upfront <00:04:41.457>about <00:04:41.613>that <00:04:41.768>because <00:04:41.924>it's <00:04:42.080>real.

76
00:04:43.500 --> 00:04:47.280
<00:04:43.500>We <00:04:43.675>solve <00:04:43.850>it <00:04:44.025>by <00:04:44.200>layer. <00:04:44.375>At <00:04:44.550>the <00:04:44.725>infrastructure <00:04:44.900>layer,

77
00:04:47.580 --> 00:04:48.940
<00:04:47.580>one <00:04:47.885>tool, <00:04:48.192>one <00:04:48.498>job <00:04:48.804>wins.

78
00:04:49.780 --> 00:04:52.080
<00:04:49.780>You'll <00:04:50.059>see <00:04:50.340>five <00:04:50.619>sharp <00:04:50.900>single-purpose <00:04:51.179>tools.

79
00:04:52.580 --> 00:04:55.380
<00:04:52.580>At <00:04:52.755>the <00:04:52.930>data <00:04:53.104>layer, <00:04:53.280>single <00:04:53.455>source <00:04:53.630>of <00:04:53.805>truth <00:04:53.979>wins.

80
00:04:56.040 --> 00:04:59.780
<00:04:56.040>And <00:04:56.180>when <00:04:56.320>the <00:04:56.460>two <00:04:56.600>collide, <00:04:56.740>we'd <00:04:56.880>rather <00:04:57.020>run <00:04:57.160>one <00:04:57.300>store <00:04:57.440>doing

81
00:04:59.840 --> 00:05:03.440
<00:04:59.840>several <00:04:59.980>jobs <00:05:00.120>than <00:05:00.260>several <00:05:00.400>stores <00:05:00.540>we <00:05:00.680>have <00:05:00.820>to <00:05:00.960>keep <00:05:01.100>in <00:05:01.240>sync.

82
00:05:03.910 --> 00:05:06.800
<00:05:03.910>And <00:05:04.110>the <00:05:04.310>Unix <00:05:04.510>idea <00:05:04.710>doesn't <00:05:04.910>appear <00:05:05.110>inside <00:05:05.310>Postgres.

83
00:05:06.880 --> 00:05:10.020
<00:05:06.880>It <00:05:07.080>just <00:05:07.280>drops <00:05:07.480>a <00:05:07.680>level <00:05:07.880>into <00:05:08.080>small <00:05:08.280>composable

84
00:05:10.180 --> 00:05:14.140
<00:05:10.180>SQL <00:05:10.355>functions <00:05:10.530>that <00:05:10.705>each <00:05:10.880>do <00:05:11.055>one <00:05:11.230>thing <00:05:11.405>and <00:05:11.580>snap

85
00:05:14.200 --> 00:05:18.000
<00:05:14.200>together. <00:05:14.355>So <00:05:14.511>it's <00:05:14.666>less <00:05:14.822>Postgres <00:05:14.977>does <00:05:15.133>everything <00:05:15.288>and <00:05:15.444>more <00:05:15.599>we

86
00:05:18.020 --> 00:05:21.760
<00:05:18.020>move <00:05:18.175>the <00:05:18.331>composition <00:05:18.486>from <00:05:18.642>the <00:05:18.797>process <00:05:18.953>boundary <00:05:19.108>down <00:05:19.264>into <00:05:19.419>the

87
00:05:21.800 --> 00:05:22.300
database.

88
00:05:23.500 --> 00:05:26.940
<00:05:23.500>From <00:05:23.733>those <00:05:23.966>two <00:05:24.200>bets <00:05:24.433>on <00:05:24.666>infrastructure, <00:05:24.900>the

89
00:05:26.980 --> 00:05:30.780
<00:05:26.980>HashiCorp <00:05:27.180>tool <00:05:27.380>chain. <00:05:27.580>On <00:05:27.780>data, <00:05:27.980>Postgres <00:05:28.180>and <00:05:28.380>PostGIS

90
00:05:30.880 --> 00:05:34.380
<00:05:30.880>as <00:05:31.020>the <00:05:31.159>source <00:05:31.300>of <00:05:31.440>truth <00:05:31.580>with <00:05:31.719>logic <00:05:31.860>pushed <00:05:32.000>to <00:05:32.140>the <00:05:32.280>data,

91
00:05:34.640 --> 00:05:38.560
<00:05:34.640>not <00:05:34.767>the <00:05:34.894>other <00:05:35.021>way <00:05:35.149>around. <00:05:35.276>Those <00:05:35.403>two <00:05:35.530>bets <00:05:35.658>are <00:05:35.785>the <00:05:35.912>two <00:05:36.039>halves

92
00:05:38.640 --> 00:05:41.120
<00:05:38.640>of <00:05:38.873>this <00:05:39.106>talk. <00:05:39.340>There's <00:05:39.573>the <00:05:39.806>punchline <00:05:40.039>to

93
00:05:41.160 --> 00:05:45.080
<00:05:41.159>preview. <00:05:41.439>A <00:05:41.719>surprising <00:05:41.999>amount <00:05:42.280>of <00:05:42.559>what

94
00:05:45.120 --> 00:05:45.810
<00:05:45.120>people <00:05:45.741>would

95
00:05:46.620 --> 00:05:50.460
<00:05:46.620>build <00:05:46.747>whole <00:05:46.874>services <00:05:47.001>for <00:05:47.129>can <00:05:47.256>instead <00:05:47.383>be <00:05:47.510>a <00:05:47.638>few <00:05:47.765>hundred <00:05:47.892>lines <00:05:48.020>of

96
00:05:50.500 --> 00:05:52.800
<00:05:50.500>SQL <00:05:50.780>living <00:05:51.060>next <00:05:51.340>to <00:05:51.620>the <00:05:51.900>data.

97
00:05:56.660 --> 00:06:00.380
<00:05:56.659>The <00:05:56.787>infrastructure <00:05:56.914>part <00:05:57.041>will <00:05:57.169>be <00:05:57.296>a <00:05:57.423>short <00:05:57.550>section, <00:05:57.678>warm <00:05:57.805>up <00:05:57.932>for <00:05:58.059>the

98
00:06:00.440 --> 00:06:02.420
<00:06:00.440>philosophy <00:06:00.719>behind <00:06:01.000>the <00:06:01.280>data <00:06:01.560>deep <00:06:01.840>dives.

99
00:06:02.910 --> 00:06:06.676
<00:06:02.910>The <00:06:03.085>claim <00:06:03.260>You <00:06:03.435>can <00:06:03.610>run <00:06:03.785>real <00:06:03.960>production <00:06:04.135>infrastructure <00:06:04.310>with

100
00:06:06.756 --> 00:06:10.076
<00:06:06.756>a <00:06:06.931>few <00:06:07.106>small <00:06:07.280>sharp <00:06:07.455>tools <00:06:07.631>instead <00:06:07.806>of <00:06:07.981>one <00:06:08.155>large

101
00:06:10.116 --> 00:06:10.976
platform.

102
00:06:13.376 --> 00:06:15.376
<00:06:13.376>This <00:06:13.575>table <00:06:13.775>is <00:06:13.976>the <00:06:14.176>life <00:06:14.376>cycle <00:06:14.575>of <00:06:14.775>workload.

103
00:06:15.836 --> 00:06:18.176
<00:06:15.836>Packer <00:06:16.069>bakes <00:06:16.302>a <00:06:16.536>known <00:06:16.769>good <00:06:17.002>image. <00:06:17.236>So

104
00:06:21.036 --> 00:06:23.136
<00:06:21.036>servers <00:06:21.386>are <00:06:21.736>immutable <00:06:22.086>and <00:06:22.436>disposable.

105
00:06:23.656 --> 00:06:27.156
<00:06:23.656>No <00:06:23.856>SSH-ing <00:06:24.056>in <00:06:24.256>and <00:06:24.456>fix, <00:06:24.656>no <00:06:24.856>drift. <00:06:25.056>Terraform

106
00:06:27.176 --> 00:06:30.796
<00:06:27.176>describes <00:06:27.331>the <00:06:27.487>cloud <00:06:27.642>as <00:06:27.798>code, <00:06:27.953>and <00:06:28.109>its <00:06:28.264>state <00:06:28.420>file <00:06:28.575>is

107
00:06:30.936 --> 00:06:34.355
<00:06:30.936>itself <00:06:31.111>a <00:06:31.286>single <00:06:31.460>source <00:06:31.635>of <00:06:31.811>truth <00:06:31.986>for <00:06:32.161>what <00:06:32.335>is

108
00:06:34.476 --> 00:06:37.336
<00:06:34.476>deployed. <00:06:34.826>Nomad <00:06:35.176>schedules <00:06:35.526>our <00:06:35.876>workloads.

109
00:06:37.576 --> 00:06:41.316
<00:06:37.576>Consul <00:06:37.731>lets <00:06:37.887>services <00:06:38.042>find <00:06:38.198>each <00:06:38.353>other <00:06:38.509>and <00:06:38.664>routes <00:06:38.820>around <00:06:38.976>unhealthy

110
00:06:41.396 --> 00:06:45.376
<00:06:41.396>ones. <00:06:41.596>And <00:06:41.796>Vault <00:06:41.996>issues <00:06:42.196>short-lived <00:06:42.396>secrets <00:06:42.596>and <00:06:42.796>credentials.

111
00:06:46.476 --> 00:06:49.996
<00:06:46.476>The <00:06:46.709>middle <00:06:46.942>point, <00:06:47.176>five <00:06:47.409>single-purpose <00:06:47.642>binaries, <00:06:47.876>each

112
00:06:50.096 --> 00:06:52.376
<00:06:50.096>replaceable, <00:06:50.376>each <00:06:50.656>with <00:06:50.936>a <00:06:51.216>clean <00:06:51.496>boundary.

113
00:06:52.836 --> 00:06:55.396
<00:06:52.836>That's <00:06:53.069>the <00:06:53.302>Unix <00:06:53.536>philosophy <00:06:53.769>applied <00:06:54.002>to <00:06:54.236>operations.

114
00:06:56.066 --> 00:06:59.676
<00:06:56.066>And <00:06:56.206>the <00:06:56.346>whole <00:06:56.486>platform <00:06:56.626>is <00:06:56.766>described <00:06:56.906>in <00:06:57.046>version <00:06:57.186>control, <00:06:57.326>so <00:06:57.466>it's

115
00:06:59.736 --> 00:07:03.656
<00:06:59.736>reproducible <00:07:00.086>and <00:07:00.436>auditable <00:07:00.786>for <00:07:01.135>free.

116
00:07:06.336 --> 00:07:09.816
<00:07:06.336>To <00:07:06.476>be <00:07:06.616>fair, <00:07:06.756>Kubernetes <00:07:06.896>is <00:07:07.036>the <00:07:07.176>right <00:07:07.316>call <00:07:07.456>at <00:07:07.596>different <00:07:07.736>scale,

117
00:07:10.716 --> 00:07:14.336
<00:07:10.716>and <00:07:10.871>its <00:07:11.027>ecosystem <00:07:11.182>is <00:07:11.338>unmatched. <00:07:11.493>But <00:07:11.649>its <00:07:11.804>complexity <00:07:11.960>is <00:07:12.116>a

118
00:07:14.396 --> 00:07:17.776
<00:07:14.396>cost <00:07:14.596>you <00:07:14.796>pay <00:07:14.996>every <00:07:15.196>day. <00:07:15.396>For <00:07:15.596>our <00:07:15.796>workload,

119
00:07:18.096 --> 00:07:21.716
<00:07:18.096>Nomad <00:07:18.223>gives <00:07:18.350>most <00:07:18.477>of <00:07:18.605>the <00:07:18.732>value <00:07:18.859>for <00:07:18.986>a <00:07:19.114>fraction <00:07:19.241>of <00:07:19.368>the <00:07:19.496>overhead.

120
00:07:22.156 --> 00:07:26.126
<00:07:22.156>A <00:07:22.311>single <00:07:22.467>binary <00:07:22.622>that <00:07:22.778>fits <00:07:22.933>in <00:07:23.089>one <00:07:23.244>person's <00:07:23.400>head, <00:07:23.556>integrating

121
00:07:26.176 --> 00:07:29.876
<00:07:26.176>natively <00:07:26.331>with <00:07:26.487>Vault <00:07:26.642>and <00:07:26.798>Consul. <00:07:26.953>So <00:07:27.109>right <00:07:27.264>tool <00:07:27.420>for <00:07:27.575>our

122
00:07:29.936 --> 00:07:30.216
scale,

123
00:07:31.296 --> 00:07:32.936
<00:07:31.296>not <00:07:31.646>a <00:07:31.996>knock <00:07:32.346>on <00:07:32.695>Kubernetes.

124
00:07:37.096 --> 00:07:40.796
<00:07:37.096>Now, <00:07:37.236>after <00:07:37.376>this <00:07:37.516>appetizer, <00:07:37.656>on <00:07:37.796>to <00:07:37.936>the <00:07:38.076>heart <00:07:38.216>of <00:07:38.356>this <00:07:38.496>talk,

125
00:07:41.076 --> 00:07:41.736
<00:07:41.076>the <00:07:41.670>thesis.

126
00:07:42.836 --> 00:07:46.816
<00:07:42.836>For <00:07:42.963>this <00:07:43.090>half <00:07:43.217>is <00:07:43.345>instead <00:07:43.472>of <00:07:43.599>a <00:07:43.726>constellation <00:07:43.854>of <00:07:43.981>specialized <00:07:44.108>stores, <00:07:44.236>we

127
00:07:46.836 --> 00:07:50.516
<00:07:46.836>lean <00:07:47.011>on <00:07:47.186>Postgres <00:07:47.361>and <00:07:47.536>push <00:07:47.711>computation <00:07:47.886>to <00:07:48.061>where <00:07:48.236>the

128
00:07:50.556 --> 00:07:51.736
<00:07:50.556>data <00:07:51.087>already <00:07:51.618>lives.

129
00:07:54.556 --> 00:07:57.796
<00:07:54.556>The <00:07:54.789>conventional <00:07:55.022>wisdom <00:07:55.256>is <00:07:55.489>for <00:07:55.722>specialized <00:07:55.955>stores.

130
00:07:58.536 --> 00:08:02.396
<00:07:58.536>Time <00:07:58.691>series <00:07:58.847>database, <00:07:59.002>graph <00:07:59.158>database, <00:07:59.313>a <00:07:59.469>stream <00:07:59.624>processor, <00:07:59.780>and <00:07:59.936>a

131
00:08:02.456 --> 00:08:05.836
<00:08:02.456>data <00:08:02.596>lake, <00:08:02.736>and <00:08:02.876>each <00:08:03.016>maps <00:08:03.156>to <00:08:03.296>a <00:08:03.436>deep <00:08:03.576>dive <00:08:03.716>coming <00:08:03.856>up.

132
00:08:06.866 --> 00:08:08.436
<00:08:06.866>The <00:08:07.216>argument <00:08:07.566>for <00:08:07.916>one <00:08:08.265>database,

133
00:08:09.456 --> 00:08:12.016
<00:08:09.456>ACID <00:08:09.736>transactions <00:08:10.016>across <00:08:10.296>all <00:08:10.576>our <00:08:10.856>data.

134
00:08:12.496 --> 00:08:13.326
<00:08:12.496>One <00:08:12.869>backup <00:08:13.243>store,

135
00:08:14.656 --> 00:08:18.436
<00:08:14.656>one <00:08:14.831>security <00:08:15.006>model, <00:08:15.181>one <00:08:15.356>query <00:08:15.531>language, <00:08:15.706>one <00:08:15.881>thing <00:08:16.056>to

136
00:08:18.476 --> 00:08:18.976
operate.

137
00:08:21.156 --> 00:08:24.976
<00:08:21.156>We <00:08:21.331>still <00:08:21.506>use <00:08:21.681>InfluxDB <00:08:21.856>for <00:08:22.031>some <00:08:22.206>monitoring <00:08:22.381>metrics, <00:08:22.556>and

138
00:08:25.016 --> 00:08:27.656
<00:08:25.016>we <00:08:25.191>also <00:08:25.366>still <00:08:25.541>use <00:08:25.716>Redis <00:08:25.891>as <00:08:26.066>a <00:08:26.241>hot <00:08:26.416>cache.

139
00:08:28.356 --> 00:08:32.076
<00:08:28.356>But <00:08:28.531>the <00:08:28.706>single <00:08:28.881>source <00:08:29.056>of <00:08:29.231>truth <00:08:29.406>means <00:08:29.581>one <00:08:29.756>authoritative

140
00:08:32.216 --> 00:08:35.856
<00:08:32.216>store. <00:08:32.371>Postgres <00:08:32.527>is <00:08:32.682>the <00:08:32.838>truth. <00:08:32.993>The <00:08:33.149>others <00:08:33.304>are <00:08:33.460>derived <00:08:33.616>and

141
00:08:35.896 --> 00:08:36.656
disposable.

142
00:08:37.856 --> 00:08:39.276
<00:08:37.856>Now, <00:08:38.175>on <00:08:38.495>to <00:08:38.814>the <00:08:39.134>how.

143
00:08:42.136 --> 00:08:46.036
<00:08:42.135>PostGIS <00:08:42.275>is <00:08:42.415>what <00:08:42.555>lets <00:08:42.695>Postgres <00:08:42.836>be <00:08:42.976>the <00:08:43.116>source <00:08:43.256>of <00:08:43.395>truth <00:08:43.535>for

144
00:08:46.136 --> 00:08:50.016
<00:08:46.136>a <00:08:46.336>location <00:08:46.536>platform. <00:08:46.736>A <00:08:46.935>mature, <00:08:47.136>fast <00:08:47.336>spatial <00:08:47.536>engine,

145
00:08:50.376 --> 00:08:52.836
<00:08:50.376>not <00:08:50.726>a <00:08:51.076>bolt-on. <00:08:51.425>Spatial <00:08:51.776>indices

146
00:08:53.976 --> 00:08:57.836
<00:08:53.976>make <00:08:54.151>what's <00:08:54.326>near <00:08:54.501>here <00:08:54.676>or <00:08:54.851>what <00:08:55.026>is <00:08:55.201>in <00:08:55.376>this

147
00:08:57.936 --> 00:08:59.916
<00:08:57.936>area <00:08:58.286>cheap, <00:08:58.636>all <00:08:58.986>in <00:08:59.336>SQL.

148
00:09:01.456 --> 00:09:05.216
<00:09:01.456>One <00:09:01.611>detail <00:09:01.767>to <00:09:01.922>flag, <00:09:02.078>the <00:09:02.233>telemetry <00:09:02.389>table <00:09:02.544>is <00:09:02.700>partitioned <00:09:02.856>by

149
00:09:05.296 --> 00:09:06.636
<00:09:05.296>time <00:09:06.502>monthly.

150
00:09:07.465 --> 00:09:10.336
<00:09:07.465>That's <00:09:07.698>the <00:09:07.931>reality <00:09:08.165>of <00:09:08.398>high <00:09:08.631>volume <00:09:08.865>interest.

151
00:09:10.956 --> 00:09:14.696
<00:09:10.956>And <00:09:11.111>old <00:09:11.267>partitions <00:09:11.422>are <00:09:11.578>exactly <00:09:11.733>what <00:09:11.889>we <00:09:12.044>offload <00:09:12.200>to <00:09:12.356>cold

152
00:09:14.776 --> 00:09:15.476
<00:09:14.776>storage <00:09:15.406>later.

153
00:09:19.616 --> 00:09:19.956
First,

154
00:09:20.876 --> 00:09:24.696
<00:09:20.876>what <00:09:21.031>Neon <00:09:21.187>actually <00:09:21.342>is, <00:09:21.498>a <00:09:21.653>cloud <00:09:21.809>provider <00:09:21.964>of <00:09:22.120>managed <00:09:22.276>serverless

155
00:09:24.796 --> 00:09:25.376
Postgres.

156
00:09:26.276 --> 00:09:30.036
<00:09:26.276>Its <00:09:26.450>defining <00:09:26.626>trick <00:09:26.800>is <00:09:26.976>separating <00:09:27.151>storage <00:09:27.325>from <00:09:27.501>compute, <00:09:27.675>and

157
00:09:30.056 --> 00:09:33.836
<00:09:30.056>that's <00:09:30.231>what <00:09:30.406>makes <00:09:30.581>our <00:09:30.756>single <00:09:30.931>source <00:09:31.106>of <00:09:31.281>truth <00:09:31.456>practically

158
00:09:33.936 --> 00:09:37.616
<00:09:33.936>rather <00:09:34.111>than <00:09:34.286>aspirational. <00:09:34.461>The <00:09:34.636>classic <00:09:34.811>pain <00:09:34.986>of <00:09:35.161>a <00:09:35.336>central

159
00:09:37.656 --> 00:09:41.596
<00:09:37.656>database <00:09:37.811>is <00:09:37.967>developing <00:09:38.122>and <00:09:38.278>testing <00:09:38.433>against <00:09:38.589>it <00:09:38.744>without <00:09:38.900>stepping <00:09:39.055>on

160
00:09:41.636 --> 00:09:44.656
<00:09:41.636>each <00:09:41.915>other. <00:09:42.195>Neon <00:09:42.476>solves <00:09:42.756>exactly <00:09:43.036>that.

161
00:09:44.776 --> 00:09:47.516
<00:09:44.776>A <00:09:45.055>branch <00:09:45.335>is <00:09:45.616>a <00:09:45.896>new <00:09:46.175>instant

162
00:09:47.776 --> 00:09:51.536
<00:09:47.776>copy-on-write <00:09:48.009>clone, <00:09:48.242>so <00:09:48.476>every <00:09:48.709>pull <00:09:48.942>request <00:09:49.175>or

163
00:09:51.676 --> 00:09:55.596
<00:09:51.676>test <00:09:51.851>run <00:09:52.026>gets <00:09:52.201>its <00:09:52.376>own <00:09:52.551>real <00:09:52.726>production <00:09:52.901>shape <00:09:53.076>database.

164
00:09:57.556 --> 00:10:01.156
<00:09:57.556>Now, <00:09:57.836>there's <00:09:58.116>a <00:09:58.396>caveat <00:09:58.676>because <00:09:58.956>it's

165
00:10:01.236 --> 00:10:04.736
<00:10:01.236>one <00:10:01.516>of <00:10:01.795>my <00:10:02.076>key <00:10:02.356>takeaways. <00:10:02.636>Cloud

166
00:10:04.796 --> 00:10:07.056
<00:10:04.796>services <00:10:05.146>are <00:10:05.496>a <00:10:05.846>double-edged <00:10:06.196>sword.

167
00:10:07.376 --> 00:10:11.216
<00:10:07.376>The <00:10:07.516>convenience <00:10:07.656>and <00:10:07.795>scale <00:10:07.935>come <00:10:08.076>with <00:10:08.216>a <00:10:08.356>real <00:10:08.496>bill <00:10:08.636>and <00:10:08.776>usually

168
00:10:11.276 --> 00:10:11.996
<00:10:11.276>vendor <00:10:11.924>login.

169
00:10:12.876 --> 00:10:14.956
<00:10:12.876>Neon <00:10:13.156>is <00:10:13.435>a <00:10:13.716>managed <00:10:13.996>dependency, <00:10:14.276>yes,

170
00:10:15.876 --> 00:10:19.476
<00:10:15.876>but <00:10:16.076>it's <00:10:16.276>still <00:10:16.476>Postgres. <00:10:16.675>No <00:10:16.876>proprietary <00:10:17.076>query <00:10:17.276>language,

171
00:10:19.916 --> 00:10:23.236
<00:10:19.916>so <00:10:20.032>the <00:10:20.149>login <00:10:20.266>is <00:10:20.382>at <00:10:20.499>the <00:10:20.616>ops <00:10:20.732>layer, <00:10:20.849>not <00:10:20.966>on <00:10:21.082>the <00:10:21.199>data <00:10:21.316>layer.

172
00:10:23.875 --> 00:10:27.416
<00:10:23.875>You <00:10:23.991>can <00:10:24.108>always <00:10:24.225>leave <00:10:24.341>with <00:10:24.458>a <00:10:24.575>pg <00:10:24.691>dump <00:10:24.808>if <00:10:24.925>you <00:10:25.041>have <00:10:25.158>the <00:10:25.275>time.

173
00:10:31.356 --> 00:10:35.336
<00:10:31.356>Four <00:10:31.556>concrete, <00:10:31.756>slightly <00:10:31.956>surprising <00:10:32.156>samples, <00:10:32.356>each <00:10:32.556>a <00:10:32.756>thing

174
00:10:35.436 --> 00:10:39.136
<00:10:35.436>people <00:10:35.591>usually <00:10:35.747>stand <00:10:35.902>up <00:10:36.058>a <00:10:36.213>separate <00:10:36.369>service <00:10:36.524>for, <00:10:36.680>and <00:10:36.836>we

175
00:10:39.176 --> 00:10:39.556
didn't.

176
00:10:41.376 --> 00:10:44.336
<00:10:41.376>The <00:10:41.531>through <00:10:41.687>line <00:10:41.842>is <00:10:41.998>not <00:10:42.153>everything <00:10:42.309>lives <00:10:42.464>in <00:10:42.620>the <00:10:42.776>database.

177
00:10:44.676 --> 00:10:48.196
<00:10:44.676>The <00:10:44.831>first <00:10:44.987>three <00:10:45.142>push <00:10:45.298>logic <00:10:45.453>into <00:10:45.609>Postgres, <00:10:45.764>but <00:10:45.920>the <00:10:46.076>fourth

178
00:10:48.516 --> 00:10:51.856
<00:10:48.516>does <00:10:48.671>the <00:10:48.827>opposite <00:10:48.982>and <00:10:49.138>pushes <00:10:49.293>cold <00:10:49.449>data <00:10:49.604>out <00:10:49.760>to <00:10:49.915>S3.

179
00:10:52.956 --> 00:10:55.896
<00:10:52.956>What <00:10:53.131>unifies <00:10:53.306>them <00:10:53.481>is <00:10:53.656>the <00:10:53.831>interface, <00:10:54.006>not <00:10:54.181>the <00:10:54.356>location.

180
00:10:56.136 --> 00:10:57.896
<00:10:56.136>It's <00:10:56.486>SQL <00:10:56.836>the <00:10:57.185>whole <00:10:57.536>way.

181
00:10:59.096 --> 00:11:01.616
<00:10:59.096>Two <00:10:59.251>of <00:10:59.407>them <00:10:59.562>are <00:10:59.718>open-sourced. <00:10:59.873>Links <00:11:00.029>are <00:11:00.184>at <00:11:00.340>the <00:11:00.496>end.

182
00:11:04.456 --> 00:11:06.556
<00:11:04.456>This <00:11:04.656>is <00:11:04.856>the <00:11:05.056>real <00:11:05.256>tracker <00:11:05.456>with <00:11:05.656>Swiss <00:11:05.856>terrain.

183
00:11:07.116 --> 00:11:09.956
<00:11:07.116>One <00:11:07.271>line's <00:11:07.427>the <00:11:07.582>raw <00:11:07.738>GPS, <00:11:07.893>the <00:11:08.049>other <00:11:08.204>the <00:11:08.360>filtered <00:11:08.516>output.

184
00:11:10.316 --> 00:11:13.576
<00:11:10.316>Noisy <00:11:10.596>positions <00:11:10.876>quietly <00:11:11.156>affect <00:11:11.436>everything <00:11:11.716>downstream.

185
00:11:13.876 --> 00:11:17.436
<00:11:13.876>Distance <00:11:14.109>totals, <00:11:14.342>geofence <00:11:14.576>triggers, <00:11:14.809>trip <00:11:15.042>detection, <00:11:15.276>range

186
00:11:17.516 --> 00:11:18.076
analytics.

187
00:11:18.956 --> 00:11:22.916
<00:11:18.956>The <00:11:19.096>standard <00:11:19.236>fix <00:11:19.376>is <00:11:19.516>a <00:11:19.656>Kalman <00:11:19.796>filter, <00:11:19.936>the <00:11:20.076>classic <00:11:20.216>recursive <00:11:20.356>estimator

188
00:11:23.556 --> 00:11:27.516
<00:11:23.556>that <00:11:23.672>predicts <00:11:23.789>where <00:11:23.906>the <00:11:24.022>vehicle <00:11:24.139>should <00:11:24.256>be <00:11:24.372>and <00:11:24.489>then <00:11:24.606>corrects <00:11:24.722>with <00:11:24.839>the <00:11:24.956>new

189
00:11:27.616 --> 00:11:30.976
<00:11:27.616>noisy <00:11:27.771>measurement, <00:11:27.927>weighting <00:11:28.082>each <00:11:28.238>by <00:11:28.393>how <00:11:28.549>much <00:11:28.704>you <00:11:28.860>trust <00:11:29.016>it.

190
00:11:31.385 --> 00:11:33.016
<00:11:31.385>The <00:11:31.735>conventional <00:11:32.085>place <00:11:32.435>to <00:11:32.785>run

191
00:11:33.856 --> 00:11:36.956
<00:11:33.856>it <00:11:34.031>is <00:11:34.206>a <00:11:34.381>streaming <00:11:34.556>service <00:11:34.731>between <00:11:34.906>ingest <00:11:35.081>and <00:11:35.256>storage.

192
00:11:37.596 --> 00:11:38.316
<00:11:37.596>Hold <00:11:37.920>that <00:11:38.244>thought.

193
00:11:40.276 --> 00:11:42.636
<00:11:40.276>The <00:11:40.626>streaming <00:11:40.976>service <00:11:41.325>is <00:11:41.675>stateful.

194
00:11:43.096 --> 00:11:46.476
<00:11:43.096>It <00:11:43.271>has <00:11:43.446>to <00:11:43.621>remember <00:11:43.796>each <00:11:43.971>device's <00:11:44.146>last <00:11:44.321>estimate <00:11:44.496>and

195
00:11:46.656 --> 00:11:50.536
<00:11:46.656>uncertainty, <00:11:46.856>and <00:11:47.055>stateful <00:11:47.256>streaming <00:11:47.455>infrastructure <00:11:47.656>is <00:11:47.856>exactly <00:11:48.055>the

196
00:11:50.576 --> 00:11:53.376
<00:11:50.576>kind <00:11:50.751>of <00:11:50.926>thing <00:11:51.101>that <00:11:51.276>needs <00:11:51.451>attention <00:11:51.626>at <00:11:51.801>awkward <00:11:51.976>hours.

197
00:11:55.556 --> 00:11:58.576
<00:11:55.556>Our <00:11:55.756>move, <00:11:55.956>implement <00:11:56.156>the <00:11:56.356>Kalman <00:11:56.556>step <00:11:56.756>as <00:11:56.956>a

198
00:11:59.156 --> 00:12:03.116
<00:11:59.156>PLPG <00:11:59.389>SQL <00:11:59.622>function <00:11:59.856>and <00:12:00.089>store <00:12:00.322>the <00:12:00.555>per-device

199
00:12:03.156 --> 00:12:06.956
<00:12:03.156>state, <00:12:03.356>last <00:12:03.555>estimate <00:12:03.756>plus <00:12:03.955>covariance <00:12:04.156>right <00:12:04.356>in <00:12:04.555>the

200
00:12:06.996 --> 00:12:07.736
<00:12:06.996>device's <00:12:07.662>row.

201
00:12:08.624 --> 00:12:12.544
<00:12:08.624>The <00:12:08.779>data <00:12:08.935>that <00:12:09.090>needs <00:12:09.246>cleaning <00:12:09.401>and <00:12:09.557>the <00:12:09.712>code <00:12:09.868>that <00:12:10.024>cleans

202
00:12:12.584 --> 00:12:12.684
it

203
00:12:14.204 --> 00:12:18.084
<00:12:14.204>live <00:12:14.343>in <00:12:14.483>the <00:12:14.623>same <00:12:14.763>place. <00:12:14.904>No <00:12:15.044>network <00:12:15.184>hop, <00:12:15.324>no <00:12:15.464>separate <00:12:15.603>state

204
00:12:18.144 --> 00:12:22.064
<00:12:18.144>store, <00:12:18.271>nothing <00:12:18.398>to <00:12:18.525>sync. <00:12:18.653>This <00:12:18.780>is <00:12:18.907>push <00:12:19.034>logic <00:12:19.162>to <00:12:19.289>the <00:12:19.416>data <00:12:19.544>made

205
00:12:22.124 --> 00:12:22.644
concrete.

206
00:12:26.984 --> 00:12:30.824
<00:12:26.984>The <00:12:27.139>no <00:12:27.295>math <00:12:27.450>version. <00:12:27.606>Two <00:12:27.761>boxes <00:12:27.917>predict, <00:12:28.072>carry <00:12:28.228>the <00:12:28.384>last

207
00:12:30.904 --> 00:12:34.404
<00:12:30.904>estimate <00:12:31.044>forward, <00:12:31.184>and <00:12:31.324>let <00:12:31.464>the <00:12:31.604>uncertainty <00:12:31.744>grow <00:12:31.884>with <00:12:32.024>the <00:12:32.164>time <00:12:32.304>gap.

208
00:12:35.064 --> 00:12:38.564
<00:12:35.064>More <00:12:35.238>time <00:12:35.414>elapse, <00:12:35.589>less <00:12:35.764>sure <00:12:35.939>where <00:12:36.113>the <00:12:36.289>vehicle <00:12:36.464>is.

209
00:12:39.264 --> 00:12:42.964
<00:12:39.264>In <00:12:39.404>the <00:12:39.544>update <00:12:39.684>step, <00:12:39.824>pull <00:12:39.964>the <00:12:40.104>estimates <00:12:40.244>toward <00:12:40.384>the <00:12:40.524>new <00:12:40.664>GPS

210
00:12:42.984 --> 00:12:46.384
<00:12:42.984>fix, <00:12:43.139>trusting <00:12:43.295>a <00:12:43.450>good <00:12:43.606>fix <00:12:43.761>more <00:12:43.917>than <00:12:44.072>a <00:12:44.228>poor <00:12:44.384>one.

211
00:12:46.884 --> 00:12:50.264
<00:12:46.884>And <00:12:47.039>that <00:12:47.195>trust <00:12:47.350>is <00:12:47.506>exactly <00:12:47.661>what <00:12:47.817>the <00:12:47.972>GPS <00:12:48.128>quality <00:12:48.284>value

212
00:12:50.564 --> 00:12:50.944
feeds.

213
00:12:52.824 --> 00:12:53.544
<00:12:52.824>The <00:12:53.472>ellipses

214
00:12:55.424 --> 00:12:56.684
<00:12:55.424>make <00:12:55.991>it <00:12:56.558>visceral.

215
00:12:57.584 --> 00:12:59.864
<00:12:57.584>Each <00:12:57.934>measurement <00:12:58.284>shrinks <00:12:58.634>the <00:12:58.984>uncertainty.

216
00:13:00.564 --> 00:13:01.804
<00:13:00.564>With <00:13:00.935>none, <00:13:01.308>it <00:13:01.680>grows.

217
00:13:04.444 --> 00:13:08.203
<00:13:04.444>Predict, <00:13:04.618>correct, <00:13:04.794>repeat, <00:13:04.968>and <00:13:05.144>the <00:13:05.319>whole <00:13:05.493>loop <00:13:05.669>is <00:13:05.843>just

218
00:13:08.284 --> 00:13:11.264
<00:13:08.284>one <00:13:08.517>SQL <00:13:08.750>function <00:13:08.984>called <00:13:09.217>once <00:13:09.450>per <00:13:09.684>point.

219
00:13:13.724 --> 00:13:15.744
<00:13:13.724>There's <00:13:13.924>two <00:13:14.124>ways <00:13:14.324>to <00:13:14.524>run <00:13:14.724>the <00:13:14.924>same <00:13:15.124>filter.

220
00:13:16.084 --> 00:13:19.064
<00:13:16.084>Online <00:13:16.317>means <00:13:16.550>filtering <00:13:16.784>as <00:13:17.017>that <00:13:17.250>data <00:13:17.483>arrives.

221
00:13:19.344 --> 00:13:23.184
<00:13:19.344>One <00:13:19.484>upsert <00:13:19.624>reads <00:13:19.764>the <00:13:19.904>device <00:13:20.044>state, <00:13:20.184>runs <00:13:20.324>a <00:13:20.464>step, <00:13:20.604>writes <00:13:20.744>back

222
00:13:23.204 --> 00:13:26.774
<00:13:23.204>the <00:13:23.359>state, <00:13:23.515>and <00:13:23.670>the <00:13:23.826>filtered <00:13:23.981>position <00:13:24.137>in <00:13:24.292>a <00:13:24.448>single <00:13:24.603>transaction,

223
00:13:27.264 --> 00:13:29.624
<00:13:27.264>so <00:13:27.419>you <00:13:27.575>can <00:13:27.730>never <00:13:27.886>have <00:13:28.041>a <00:13:28.197>position <00:13:28.352>without <00:13:28.508>its <00:13:28.664>state.

224
00:13:30.184 --> 00:13:34.044
<00:13:30.184>Data <00:13:30.384>is <00:13:30.584>usable <00:13:30.784>immediately. <00:13:30.983>No <00:13:31.184>nightly <00:13:31.384>cleaning <00:13:31.584>job.

225
00:13:34.284 --> 00:13:37.984
<00:13:34.284>The <00:13:34.424>cost <00:13:34.564>is <00:13:34.704>work <00:13:34.843>on <00:13:34.984>the <00:13:35.124>hot <00:13:35.264>insert <00:13:35.404>path, <00:13:35.544>which <00:13:35.684>I'll

226
00:13:38.004 --> 00:13:38.924
<00:13:38.004>quantify <00:13:38.832>next.

227
00:13:40.244 --> 00:13:42.884
<00:13:40.244>Offline <00:13:40.710>means <00:13:41.177>reprocessed <00:13:41.644>history.

228
00:13:43.284 --> 00:13:47.164
<00:13:43.284>Say <00:13:43.459>you <00:13:43.634>retune <00:13:43.809>the <00:13:43.984>parameter. <00:13:44.159>The <00:13:44.334>filter <00:13:44.509>is <00:13:44.684>inherently

229
00:13:47.244 --> 00:13:47.884
sequential,

230
00:13:48.964 --> 00:13:51.184
<00:13:48.964>which <00:13:49.314>set-based <00:13:49.664>SQL <00:13:50.013>doesn't <00:13:50.363>love.

231
00:13:51.684 --> 00:13:54.044
<00:13:51.684>So <00:13:51.964>there <00:13:52.243>are <00:13:52.524>two <00:13:52.804>idiomatic <00:13:53.084>options.

232
00:13:54.484 --> 00:13:58.164
<00:13:54.484>A <00:13:54.659>recursive <00:13:54.834>CTE, <00:13:55.009>which <00:13:55.184>is <00:13:55.359>great <00:13:55.534>for <00:13:55.709>debugging <00:13:55.884>because

233
00:13:58.264 --> 00:14:01.944
<00:13:58.264>every <00:13:58.439>intermediate <00:13:58.614>state <00:13:58.789>is <00:13:58.964>visible, <00:13:59.139>or <00:13:59.314>a <00:13:59.489>custom <00:13:59.664>aggregate

234
00:14:02.024 --> 00:14:05.914
<00:14:02.024>over <00:14:02.179>a <00:14:02.335>window, <00:14:02.490>which <00:14:02.646>reads <00:14:02.801>like <00:14:02.957>normal <00:14:03.112>analytics <00:14:03.268>SQL <00:14:03.424>and

235
00:14:05.944 --> 00:14:09.564
<00:14:05.944>is <00:14:06.118>faster. <00:14:06.294>Both <00:14:06.468>reuse <00:14:06.644>the <00:14:06.819>exact <00:14:06.993>same <00:14:07.169>Kalman <00:14:07.343>step.

236
00:14:11.604 --> 00:14:15.164
<00:14:11.604>Let <00:14:11.759>me <00:14:11.915>be <00:14:12.070>straight. <00:14:12.226>Online <00:14:12.381>filtering <00:14:12.537>is <00:14:12.692>real <00:14:12.848>work, <00:14:13.004>about

237
00:14:15.384 --> 00:14:18.993
<00:14:15.384>60% <00:14:15.664>more <00:14:15.944>latency, <00:14:16.224>throughput <00:14:16.504>dropping <00:14:16.784>from

238
00:14:19.084 --> 00:14:22.804
<00:14:19.084>117 <00:14:19.284>to <00:14:19.483>74 <00:14:19.684>transaction <00:14:19.883>per <00:14:20.084>second <00:14:20.284>per <00:14:20.483>client.

239
00:14:23.884 --> 00:14:27.144
<00:14:23.884>But <00:14:24.024>connect <00:14:24.164>that <00:14:24.304>to <00:14:24.444>the <00:14:24.584>real <00:14:24.724>load. <00:14:24.864>Our <00:14:25.004>peak <00:14:25.144>is <00:14:25.284>around

240
00:14:27.324 --> 00:14:31.124
<00:14:27.324>150 <00:14:27.524>records <00:14:27.723>per <00:14:27.924>second, <00:14:28.123>and <00:14:28.324>74 <00:14:28.524>is <00:14:28.723>per

241
00:14:31.224 --> 00:14:35.094
<00:14:31.224>single <00:14:31.424>client, <00:14:31.624>so <00:14:31.824>two <00:14:32.024>or <00:14:32.224>three <00:14:32.424>concurrent <00:14:32.624>connections

242
00:14:35.744 --> 00:14:36.844
<00:14:35.744>cleared <00:14:36.239>with <00:14:36.734>headroom.

243
00:14:37.824 --> 00:14:41.564
<00:14:37.824>The <00:14:38.103>scary-sounding <00:14:38.383>60% <00:14:38.664>slower <00:14:38.944>is <00:14:39.223>comfortably

244
00:14:41.624 --> 00:14:42.744
<00:14:41.624>inside <00:14:42.128>our <00:14:42.632>budget.

245
00:14:43.784 --> 00:14:45.644
<00:14:43.784>That's <00:14:43.984>the <00:14:44.184>point <00:14:44.384>of <00:14:44.584>having <00:14:44.784>the <00:14:44.984>real <00:14:45.184>number.

246
00:14:45.884 --> 00:14:49.664
<00:14:45.884>We <00:14:46.039>measured <00:14:46.195>the <00:14:46.350>cost <00:14:46.506>against <00:14:46.661>our <00:14:46.817>actual <00:14:46.972>scale, <00:14:47.128>not <00:14:47.284>a

247
00:14:49.784 --> 00:14:51.364
<00:14:49.784>hypothetical <00:14:50.484>fire <00:14:51.184>hose.

248
00:14:52.904 --> 00:14:55.924
<00:14:52.904>Offline, <00:14:53.184>the <00:14:53.464>custom <00:14:53.744>aggregate <00:14:54.024>beats <00:14:54.304>the

249
00:14:55.984 --> 00:14:59.544
<00:14:55.984>recursive <00:14:56.264>CTE <00:14:56.544>and <00:14:56.824>is <00:14:57.104>cleaner. <00:14:57.384>So

250
00:15:00.064 --> 00:15:03.924
<00:15:00.064>aggregate <00:15:00.204>for <00:15:00.343>batch, <00:15:00.483>recursive <00:15:00.623>CTE <00:15:00.764>when <00:15:00.904>you <00:15:01.044>need <00:15:01.184>to <00:15:01.324>see <00:15:01.464>the

251
00:15:03.984 --> 00:15:04.424
steps.

252
00:15:08.184 --> 00:15:12.053
<00:15:08.184>So <00:15:08.358>the <00:15:08.534>takeaway <00:15:08.709>for <00:15:08.884>this <00:15:09.059>first <00:15:09.233>deep <00:15:09.409>dive, <00:15:09.584>we've

253
00:15:12.104 --> 00:15:15.684
<00:15:12.104>removed <00:15:12.304>an <00:15:12.504>entire <00:15:12.704>class <00:15:12.904>of <00:15:13.104>infrastructure, <00:15:13.304>a <00:15:13.504>stateful

254
00:15:16.624 --> 00:15:20.264
<00:15:16.624>stream <00:15:16.764>processor, <00:15:16.904>and <00:15:17.044>replaced <00:15:17.184>it <00:15:17.324>with <00:15:17.464>SQL <00:15:17.604>next <00:15:17.744>to <00:15:17.884>the <00:15:18.024>data.

255
00:15:20.854 --> 00:15:24.084
<00:15:20.854>The <00:15:21.029>trade <00:15:21.204>we <00:15:21.379>accepted, <00:15:21.554>some <00:15:21.729>load <00:15:21.904>on <00:15:22.079>the <00:15:22.254>database,

256
00:15:24.884 --> 00:15:28.584
<00:15:24.884>and <00:15:25.164>PLPG <00:15:25.444>SQL <00:15:25.724>isn't <00:15:26.004>everyone's <00:15:26.284>favorite

257
00:15:28.644 --> 00:15:32.484
<00:15:28.644>language. <00:15:28.844>Cheaper <00:15:29.044>for <00:15:29.244>us <00:15:29.444>than <00:15:29.644>operating <00:15:29.844>another <00:15:30.044>stateful

258
00:15:32.544 --> 00:15:32.984
service.

259
00:15:34.024 --> 00:15:37.084
<00:15:34.024>It's <00:15:34.164>open <00:15:34.304>source <00:15:34.444>if <00:15:34.584>you <00:15:34.724>want <00:15:34.864>to <00:15:35.004>read <00:15:35.144>the <00:15:35.284>matrix <00:15:35.424>functions.

260
00:15:40.864 --> 00:15:44.804
<00:15:40.864>Now, <00:15:40.991>the <00:15:41.118>positions <00:15:41.245>are <00:15:41.373>clean, <00:15:41.500>but <00:15:41.627>what <00:15:41.754>do <00:15:41.882>we <00:15:42.009>do <00:15:42.136>with <00:15:42.264>them?

261
00:15:45.244 --> 00:15:48.914
<00:15:45.244>Here's <00:15:45.419>the <00:15:45.594>deep <00:15:45.769>dive <00:15:45.944>number <00:15:46.119>2, <00:15:46.294>turning <00:15:46.469>points <00:15:46.644>into

262
00:15:48.944 --> 00:15:49.284
trips.

263
00:15:50.504 --> 00:15:54.304
<00:15:50.504>Nobody <00:15:50.644>looks <00:15:50.784>at <00:15:50.924>raw <00:15:51.064>points. <00:15:51.204>Our <00:15:51.344>customers, <00:15:51.484>they <00:15:51.624>look <00:15:51.764>at <00:15:51.904>trips.

264
00:15:55.044 --> 00:15:58.984
<00:15:55.044>So <00:15:55.218>between <00:15:55.394>point <00:15:55.569>land <00:15:55.744>and <00:15:55.919>trips <00:15:56.093>appear, <00:15:56.269>there's <00:15:56.444>normally

265
00:15:59.004 --> 00:16:02.864
<00:15:59.004>a <00:15:59.204>stream <00:15:59.404>processing <00:15:59.604>service. <00:15:59.804>Ours <00:16:00.004>runs <00:16:00.204>inside <00:16:00.404>Postgres,

266
00:16:03.164 --> 00:16:06.884
<00:16:03.164>and <00:16:03.397>it's <00:16:03.630>almost <00:16:03.864>online. <00:16:04.097>Trips <00:16:04.330>materialize <00:16:04.564>within

267
00:16:07.004 --> 00:16:10.904
<00:16:07.004>seconds, <00:16:07.204>maybe <00:16:07.404>minutes, <00:16:07.604>of <00:16:07.804>the <00:16:08.004>data <00:16:08.204>arriving, <00:16:08.404>not

268
00:16:10.964 --> 00:16:14.764
<00:16:10.964>in <00:16:11.104>a <00:16:11.244>nightly <00:16:11.384>batch. <00:16:11.524>This <00:16:11.664>is <00:16:11.804>the <00:16:11.944>second <00:16:12.084>thing, <00:16:12.224>replacing <00:16:12.364>that

269
00:16:14.884 --> 00:16:17.944
<00:16:14.884>stream <00:16:15.084>processor. <00:16:15.284>The <00:16:15.484>Kalman <00:16:15.684>filter <00:16:15.884>was <00:16:16.084>the <00:16:16.284>first.

270
00:16:21.624 --> 00:16:21.784
So

271
00:16:22.664 --> 00:16:26.164
<00:16:22.664>two <00:16:22.897>moving <00:16:23.130>parts, <00:16:23.364>both <00:16:23.597>SQL. <00:16:23.830>Triggers <00:16:24.064>watch

272
00:16:26.244 --> 00:16:29.644
<00:16:26.244>incoming <00:16:26.419>telemetry <00:16:26.594>and <00:16:26.769>open, <00:16:26.944>extend, <00:16:27.119>or <00:16:27.294>close <00:16:27.469>a <00:16:27.644>leg

273
00:16:29.684 --> 00:16:33.064
<00:16:29.684>whenever <00:16:29.884>the <00:16:30.084>ignition <00:16:30.284>flips. <00:16:30.483>A <00:16:30.684>leg <00:16:30.884>is <00:16:31.084>one

274
00:16:33.164 --> 00:16:35.204
<00:16:33.164>continuous <00:16:33.864>ignition-on <00:16:34.564>period.

275
00:16:36.144 --> 00:16:40.104
<00:16:36.144>Then <00:16:36.299>queries <00:16:36.455>run <00:16:36.610>continuously <00:16:36.766>and <00:16:36.921>merge <00:16:37.077>consecutive <00:16:37.232>legs <00:16:37.388>into <00:16:37.544>a

276
00:16:40.144 --> 00:16:44.124
<00:16:40.144>trip, <00:16:40.344>allowing <00:16:40.544>a <00:16:40.744>configurable <00:16:40.944>gap, <00:16:41.144>so <00:16:41.344>a <00:16:41.544>quick

277
00:16:44.184 --> 00:16:46.984
<00:16:44.184>fuel <00:16:44.384>stop <00:16:44.584>doesn't <00:16:44.784>shatter <00:16:44.983>one <00:16:45.184>journey <00:16:45.384>into <00:16:45.584>several.

278
00:16:48.304 --> 00:16:51.724
<00:16:48.304>On <00:16:48.504>the <00:16:48.704>diagram, <00:16:48.904>three <00:16:49.103>legs. <00:16:49.304>The <00:16:49.504>short <00:16:49.704>stop

279
00:16:51.924 --> 00:16:52.784
<00:16:51.924>merges <00:16:52.698>legs

280
00:16:53.604 --> 00:16:56.884
<00:16:53.604>one <00:16:53.759>and <00:16:53.915>two <00:16:54.070>into <00:16:54.226>trip <00:16:54.381>A. <00:16:54.537>The <00:16:54.692>long <00:16:54.848>stop <00:16:55.004>makes

281
00:16:57.684 --> 00:17:00.163
<00:16:57.684>leg <00:16:57.964>three <00:16:58.243>its <00:16:58.524>own <00:16:58.804>trip <00:16:59.084>B.

282
00:17:01.724 --> 00:17:05.325
<00:17:01.724>Almost <00:17:01.957>online <00:17:02.190>is <00:17:02.424>honest. <00:17:02.657>It's <00:17:02.890>seconds <00:17:03.124>fresh,

283
00:17:06.325 --> 00:17:09.664
<00:17:06.325>sometimes <00:17:06.605>minutes <00:17:06.885>fresh, <00:17:07.165>at <00:17:07.444>zero <00:17:07.725>extra

284
00:17:09.724 --> 00:17:10.585
infrastructure.

285
00:17:15.504 --> 00:17:18.585
<00:17:15.503>A <00:17:15.644>trip <00:17:15.783>isn't <00:17:15.924>just <00:17:16.063>a <00:17:16.204>start <00:17:16.343>and <00:17:16.483>end. <00:17:16.623>It's <00:17:16.763>a <00:17:16.904>rich

286
00:17:18.825 --> 00:17:22.484
<00:17:18.825>pre-aggregated <00:17:19.025>record. <00:17:19.225>As <00:17:19.425>legs <00:17:19.625>close <00:17:19.825>and <00:17:20.025>trips <00:17:20.225>form,

287
00:17:22.805 --> 00:17:26.664
<00:17:22.805>the <00:17:23.005>same <00:17:23.205>queries <00:17:23.405>roll <00:17:23.605>the <00:17:23.805>metrics <00:17:24.005>that <00:17:24.205>business

288
00:17:26.805 --> 00:17:27.144
asks

289
00:17:29.464 --> 00:17:33.284
<00:17:29.464>for. <00:17:29.697>Note <00:17:29.930>the <00:17:30.164>domain-specific <00:17:30.397>ones, <00:17:30.630>plowing <00:17:30.864>and

290
00:17:33.324 --> 00:17:34.084
<00:17:33.324>PTO <00:17:34.008>time,

291
00:17:35.504 --> 00:17:38.804
<00:17:35.504>where <00:17:35.678>work <00:17:35.853>done <00:17:36.029>matters <00:17:36.204>more <00:17:36.379>than <00:17:36.553>the <00:17:36.728>distance, <00:17:36.904>and

292
00:17:39.104 --> 00:17:40.284
<00:17:39.104>geo-zone <00:17:40.166>analytics,

293
00:17:41.204 --> 00:17:43.894
<00:17:41.204>which <00:17:41.379>leans <00:17:41.553>directly <00:17:41.729>on <00:17:41.904>the <00:17:42.079>Postgres <00:17:42.254>work <00:17:42.428>from <00:17:42.604>earlier.

294
00:17:44.624 --> 00:17:47.624
<00:17:44.624>For <00:17:44.799>each <00:17:44.974>zone, <00:17:45.149>how <00:17:45.324>long <00:17:45.499>and <00:17:45.674>how <00:17:45.849>far <00:17:46.024>inside?

295
00:17:48.364 --> 00:17:52.064
<00:17:48.364>Because <00:17:48.519>it's <00:17:48.675>all <00:17:48.830>one <00:17:48.986>transaction-consistent <00:17:49.141>store, <00:17:49.297>a <00:17:49.452>trip <00:17:49.608>and <00:17:49.764>its

296
00:17:52.124 --> 00:17:55.824
<00:17:52.124>aggregates <00:17:52.357>can't <00:17:52.590>drift <00:17:52.824>apart, <00:17:53.057>and <00:17:53.290>reporting <00:17:53.524>is

297
00:17:55.944 --> 00:17:56.784
<00:17:55.944>just <00:17:56.322>a <00:17:56.700>select.

298
00:17:59.264 --> 00:18:03.104
<00:17:59.264>So <00:17:59.464>takeaway, <00:17:59.664>second <00:17:59.863>takeaway <00:18:00.063>of <00:18:00.264>this <00:18:00.464>deep <00:18:00.664>dive.

299
00:18:03.884 --> 00:18:07.584
<00:18:03.884>The <00:18:04.039>classic <00:18:04.195>streaming <00:18:04.350>component, <00:18:04.506>turning <00:18:04.661>an <00:18:04.817>event <00:18:04.972>feed <00:18:05.128>into <00:18:05.284>meaningful

300
00:18:07.724 --> 00:18:11.504
<00:18:07.724>sessions <00:18:07.864>is, <00:18:08.004>for <00:18:08.144>us, <00:18:08.283>triggers <00:18:08.424>plus <00:18:08.563>queries <00:18:08.704>next <00:18:08.843>to <00:18:08.984>the <00:18:09.124>data.

301
00:18:12.024 --> 00:18:12.684
<00:18:12.024>Seconds <00:18:12.618>fresh

302
00:18:13.992 --> 00:18:16.692
<00:18:13.992>No <00:18:14.272>broker, <00:18:14.552>no <00:18:14.831>consumer <00:18:15.111>queue. <00:18:15.392>Same

303
00:18:17.892 --> 00:18:21.492
<00:18:17.892>trade <00:18:18.047>as <00:18:18.203>before. <00:18:18.358>A <00:18:18.514>bit <00:18:18.669>of <00:18:18.825>database <00:18:18.980>load <00:18:19.136>for <00:18:19.292>one

304
00:18:21.652 --> 00:18:22.832
<00:18:21.652>fewer <00:18:22.183>moving <00:18:22.714>part.

305
00:18:25.992 --> 00:18:29.472
<00:18:25.992>On <00:18:26.132>to <00:18:26.272>the <00:18:26.412>next <00:18:26.552>deep <00:18:26.692>dive, <00:18:26.831>trailers <00:18:26.972>that <00:18:27.111>come <00:18:27.252>and <00:18:27.392>go.

306
00:18:29.992 --> 00:18:32.572
<00:18:29.992>So <00:18:30.272>vehicles <00:18:30.552>attached <00:18:30.831>to <00:18:31.111>each <00:18:31.392>other.

307
00:18:32.632 --> 00:18:35.252
<00:18:32.632>Pulling <00:18:32.865>vehicle <00:18:33.098>can <00:18:33.332>have <00:18:33.565>several <00:18:33.798>behind <00:18:34.032>it.

308
00:18:36.572 --> 00:18:39.912
<00:18:36.572>If <00:18:36.852>we <00:18:37.132>treat <00:18:37.412>trailers <00:18:37.692>like <00:18:37.972>vehicles,

309
00:18:40.272 --> 00:18:42.452
<00:18:40.272>as <00:18:40.621>the <00:18:40.972>same <00:18:41.322>object. <00:18:41.672>So

310
00:18:44.352 --> 00:18:48.152
<00:18:44.352>pulling <00:18:44.492>vehicle <00:18:44.632>can <00:18:44.772>have <00:18:44.912>several <00:18:45.052>behind <00:18:45.192>it, <00:18:45.332>and <00:18:45.472>the <00:18:45.612>chain <00:18:45.752>forms,

311
00:18:48.492 --> 00:18:51.132
<00:18:48.492>changing <00:18:48.841>constantly. <00:18:49.192>Two <00:18:49.542>hard <00:18:49.892>parts.

312
00:18:51.372 --> 00:18:54.992
<00:18:51.372>It's <00:18:51.572>temporal, <00:18:51.772>so <00:18:51.972>we <00:18:52.172>must <00:18:52.372>answer <00:18:52.572>about <00:18:52.772>any

313
00:18:55.132 --> 00:18:58.892
<00:18:55.132>past <00:18:55.332>instant, <00:18:55.532>not <00:18:55.732>just <00:18:55.932>now, <00:18:56.132>and <00:18:56.332>it's <00:18:56.532>multi-source.

314
00:18:59.332 --> 00:19:03.262
<00:18:59.332>A <00:18:59.472>driver <00:18:59.612>enters <00:18:59.752>one <00:18:59.892>coupling, <00:19:00.032>a <00:19:00.172>sender <00:19:00.312>reports <00:19:00.452>another, <00:19:00.592>and <00:19:00.732>they

315
00:19:03.332 --> 00:19:03.792
conflict.

316
00:19:05.512 --> 00:19:08.792
<00:19:05.512>So <00:19:05.712>with <00:19:05.912>some <00:19:06.111>sources <00:19:06.312>more <00:19:06.512>trustworthy <00:19:06.712>than <00:19:06.912>others.

317
00:19:10.552 --> 00:19:14.532
<00:19:10.552>This <00:19:10.679>is <00:19:10.806>naturally <00:19:10.933>a <00:19:11.061>graph <00:19:11.188>problem, <00:19:11.315>which <00:19:11.442>tempts <00:19:11.570>you <00:19:11.697>towards <00:19:11.824>a <00:19:11.952>graph

318
00:19:14.592 --> 00:19:15.152
database.

319
00:19:15.992 --> 00:19:19.472
<00:19:15.992>We'll <00:19:16.147>model <00:19:16.303>it <00:19:16.458>as <00:19:16.614>a <00:19:16.769>graph <00:19:16.925>too, <00:19:17.080>but <00:19:17.236>inside <00:19:17.392>Postgres.

320
00:19:19.792 --> 00:19:23.512
<00:19:19.792>Vehicles <00:19:19.947>are <00:19:20.103>nodes. <00:19:20.258>A <00:19:20.414>pulls <00:19:20.569>B <00:19:20.725>is <00:19:20.880>a <00:19:21.036>directed <00:19:21.192>edge.

321
00:19:26.512 --> 00:19:28.032
<00:19:26.512>Here's <00:19:26.740>how <00:19:26.967>the <00:19:27.196>graph <00:19:27.424>and <00:19:27.652>its <00:19:27.879>rule

322
00:19:28.992 --> 00:19:30.052
<00:19:28.992>look <00:19:29.469>like <00:19:29.946>together.

323
00:19:31.252 --> 00:19:34.932
<00:19:31.252>Vehicle <00:19:31.407>one <00:19:31.563>pulls <00:19:31.718>vehicle <00:19:31.874>two, <00:19:32.029>which <00:19:32.185>pulls <00:19:32.340>vehicle <00:19:32.496>three <00:19:32.652>and

324
00:19:35.012 --> 00:19:38.312
<00:19:35.012>four. <00:19:35.212>A <00:19:35.412>chain <00:19:35.611>that <00:19:35.812>branches <00:19:36.012>but <00:19:36.212>never <00:19:36.412>loops.

325
00:19:39.672 --> 00:19:43.612
<00:19:39.672>Four <00:19:39.812>rules <00:19:39.952>in <00:19:40.092>the <00:19:40.232>spec. <00:19:40.372>A <00:19:40.512>vehicle <00:19:40.652>can <00:19:40.792>pull <00:19:40.932>many <00:19:41.072>at

326
00:19:43.712 --> 00:19:47.312
<00:19:43.712>once, <00:19:43.839>but <00:19:43.966>it <00:19:44.093>is <00:19:44.221>pulled <00:19:44.348>by <00:19:44.475>at <00:19:44.602>most <00:19:44.730>one <00:19:44.857>at <00:19:44.984>a <00:19:45.112>time.

327
00:19:47.792 --> 00:19:48.612
<00:19:47.792>No <00:19:48.530>cycles.

328
00:19:49.432 --> 00:19:52.292
<00:19:49.432>Conflicting <00:19:49.712>sources <00:19:49.992>are <00:19:50.272>resolved <00:19:50.552>by <00:19:50.832>priority.

329
00:19:52.872 --> 00:19:56.472
<00:19:52.872>It's <00:19:53.072>time <00:19:53.272>variant, <00:19:53.472>because <00:19:53.672>every <00:19:53.872>edge <00:19:54.072>is <00:19:54.272>valid

330
00:19:56.572 --> 00:19:59.812
<00:19:56.572>only <00:19:56.805>over <00:19:57.038>an <00:19:57.272>interval, <00:19:57.505>and <00:19:57.738>all <00:19:57.972>forms,

331
00:20:00.372 --> 00:20:04.112
<00:20:00.372>rules <00:20:00.572>must <00:20:00.772>hold <00:20:00.972>at <00:20:01.172>every <00:20:01.372>instant. <00:20:01.572>Postgres <00:20:01.772>has

332
00:20:04.132 --> 00:20:07.592
<00:20:04.132>exactly <00:20:04.332>the <00:20:04.532>primitives <00:20:04.732>for <00:20:04.932>this. <00:20:05.132>Interval <00:20:05.332>columns, <00:20:05.532>check

333
00:20:07.612 --> 00:20:11.332
<00:20:07.612>constraints, <00:20:07.845>deferred <00:20:08.078>triggers, <00:20:08.312>recursive <00:20:08.545>CTEs, <00:20:08.778>so <00:20:09.012>we

334
00:20:11.372 --> 00:20:13.472
<00:20:11.372>don't <00:20:11.838>need <00:20:12.305>graph <00:20:12.772>database.

335
00:20:16.682 --> 00:20:20.432
<00:20:16.682>The <00:20:16.822>whole <00:20:16.962>model <00:20:17.102>is <00:20:17.242>a <00:20:17.382>table. <00:20:17.522>An <00:20:17.662>edge <00:20:17.802>carries <00:20:17.942>its <00:20:18.082>lifespan,

336
00:20:20.632 --> 00:20:24.112
<00:20:20.632>valid <00:20:20.832>from <00:20:21.032>and <00:20:21.232>valid <00:20:21.432>to, <00:20:21.632>where <00:20:21.832>null <00:20:22.032>means

337
00:20:24.172 --> 00:20:26.352
<00:20:24.172>ongoing, <00:20:24.638>and <00:20:25.105>its <00:20:25.572>priority,

338
00:20:28.392 --> 00:20:32.332
<00:20:28.392>which <00:20:28.567>source <00:20:28.742>set. <00:20:28.917>So <00:20:29.092>two <00:20:29.267>cheap <00:20:29.442>check <00:20:29.617>constraints <00:20:29.792>already

339
00:20:32.432 --> 00:20:35.512
<00:20:32.432>rule <00:20:32.632>out <00:20:32.832>self-loops <00:20:33.032>and <00:20:33.232>bad <00:20:33.432>intervals <00:20:33.632>for <00:20:33.832>free.

340
00:20:36.652 --> 00:20:40.232
<00:20:36.652>This <00:20:36.807>is <00:20:36.963>the <00:20:37.118>model <00:20:37.274>is <00:20:37.429>the <00:20:37.585>data. <00:20:37.740>No <00:20:37.896>ORM, <00:20:38.052>graph

341
00:20:40.292 --> 00:20:43.132
<00:20:40.292>abstraction, <00:20:40.572>relational <00:20:40.851>table <00:20:41.131>is <00:20:41.411>the <00:20:41.692>graph.

342
00:20:44.042 --> 00:20:47.762
<00:20:44.042>The <00:20:44.242>harder <00:20:44.442>rules, <00:20:44.641>no <00:20:44.841>cycles <00:20:45.042>and <00:20:45.242>single <00:20:45.442>incoming

343
00:20:47.812 --> 00:20:51.552
<00:20:47.812>edge, <00:20:47.939>they <00:20:48.066>need <00:20:48.193>to <00:20:48.321>look <00:20:48.448>at <00:20:48.575>other <00:20:48.702>rows, <00:20:48.830>which <00:20:48.957>is <00:20:49.084>the <00:20:49.212>next

344
00:20:51.572 --> 00:20:51.932
slide.

345
00:20:54.132 --> 00:20:56.052
<00:20:54.132>So <00:20:54.365>these <00:20:54.598>rules <00:20:54.832>live <00:20:55.065>in <00:20:55.298>the <00:20:55.532>database.

346
00:20:56.652 --> 00:21:00.382
<00:20:56.652>They <00:20:56.827>hold <00:20:57.002>no <00:20:57.177>matter <00:20:57.352>who <00:20:57.527>writes, <00:20:57.702>API <00:20:57.877>migration <00:20:58.052>or

347
00:21:00.472 --> 00:21:04.132
<00:21:00.472>manual <00:21:00.627>fix. <00:21:00.783>You <00:21:00.938>simply <00:21:01.094>cannot <00:21:01.249>get <00:21:01.405>a <00:21:01.560>cycle <00:21:01.716>into <00:21:01.872>this

348
00:21:04.152 --> 00:21:07.692
<00:21:04.152>table. <00:21:04.385>Cycle <00:21:04.618>detection <00:21:04.852>is <00:21:05.085>itself <00:21:05.318>a <00:21:05.552>recursive

349
00:21:07.752 --> 00:21:11.172
<00:21:07.752>CTE, <00:21:07.952>and <00:21:08.152>it's <00:21:08.351>interval <00:21:08.552>aware. <00:21:08.752>A <00:21:08.952>pulling <00:21:09.152>B

350
00:21:11.252 --> 00:21:14.292
<00:21:11.252>yesterday <00:21:11.407>and <00:21:11.563>B <00:21:11.718>pulling <00:21:11.874>A <00:21:12.029>today <00:21:12.185>is <00:21:12.340>not <00:21:12.496>a <00:21:12.652>cycle.

351
00:21:16.052 --> 00:21:18.992
<00:21:16.052>Deferred <00:21:16.331>is <00:21:16.611>the <00:21:16.891>subtle <00:21:17.171>important <00:21:17.452>bit.

352
00:21:19.432 --> 00:21:23.312
<00:21:19.432>Conflict <00:21:19.712>resolution <00:21:19.992>rewrites <00:21:20.272>several <00:21:20.552>edges <00:21:20.832>in

353
00:21:23.352 --> 00:21:24.152
<00:21:23.352>one <00:21:24.072>transaction,

354
00:21:25.072 --> 00:21:28.882
<00:21:25.072>so <00:21:25.227>it <00:21:25.383>may <00:21:25.538>be <00:21:25.694>briefly <00:21:25.849>passing <00:21:26.005>through <00:21:26.160>illegal <00:21:26.316>intermediate <00:21:26.472>states,

355
00:21:29.312 --> 00:21:32.712
<00:21:29.312>so <00:21:29.512>we <00:21:29.712>defer <00:21:29.911>the <00:21:30.111>check <00:21:30.312>to <00:21:30.512>commit <00:21:30.712>time.

356
00:21:33.692 --> 00:21:37.472
<00:21:33.692>The <00:21:33.847>headline <00:21:34.003>for <00:21:34.158>everyone, <00:21:34.314>the <00:21:34.469>database <00:21:34.625>itself <00:21:34.780>won't <00:21:34.936>store <00:21:35.092>an

357
00:21:37.532 --> 00:21:39.251
<00:21:37.532>impossible <00:21:38.932>state.

358
00:21:43.192 --> 00:21:47.112
<00:21:43.192>Add <00:21:43.425>edge. <00:21:43.658>This <00:21:43.892>is <00:21:44.125>the <00:21:44.358>conflict <00:21:44.592>resolution.

359
00:21:47.552 --> 00:21:51.162
<00:21:47.552>This <00:21:47.726>is <00:21:47.901>the <00:21:48.077>payoff <00:21:48.252>function. <00:21:48.427>Callers <00:21:48.601>don't <00:21:48.776>reason <00:21:48.952>about

360
00:21:51.312 --> 00:21:55.272
<00:21:51.312>overlaps. <00:21:51.467>They <00:21:51.623>assert <00:21:51.778>this <00:21:51.934>source <00:21:52.089>says <00:21:52.245>A <00:21:52.400>pull <00:21:52.556>B <00:21:52.712>over

361
00:21:55.372 --> 00:21:59.052
<00:21:55.372>this <00:21:55.572>interval <00:21:55.772>with <00:21:55.972>this <00:21:56.172>authority, <00:21:56.372>and <00:21:56.572>add <00:21:56.772>edge,

362
00:21:59.412 --> 00:22:02.172
<00:21:59.412>this <00:21:59.878>SQL <00:22:00.345>function <00:22:00.812>reconciles.

363
00:22:03.012 --> 00:22:06.792
<00:22:03.012>On <00:22:03.187>the <00:22:03.361>timeline, <00:22:03.537>the <00:22:03.712>higher <00:22:03.887>priority <00:22:04.062>edge <00:22:04.236>wins <00:22:04.412>the

364
00:22:06.852 --> 00:22:10.372
<00:22:06.852>overlap. <00:22:07.132>The <00:22:07.412>lower <00:22:07.692>priority <00:22:07.972>is <00:22:08.252>truncated.

365
00:22:10.992 --> 00:22:11.612
<00:22:10.992>Fully <00:22:11.550>covered

366
00:22:13.792 --> 00:22:17.132
<00:22:13.792>inserts <00:22:13.992>are <00:22:14.192>dropped, <00:22:14.391>and <00:22:14.591>straddling <00:22:14.792>ones <00:22:14.992>get <00:22:15.192>split.

367
00:22:18.492 --> 00:22:21.452
<00:22:18.492>That's <00:22:18.692>the <00:22:18.892>multi-source <00:22:19.091>conflict <00:22:19.292>resolution <00:22:19.492>from <00:22:19.692>rule <00:22:19.892>four,

368
00:22:22.012 --> 00:22:25.432
<00:22:22.012>automated, <00:22:22.187>and <00:22:22.361>that's <00:22:22.537>why <00:22:22.712>the <00:22:22.887>triggers <00:22:23.062>must <00:22:23.236>be <00:22:23.412>deferred,

369
00:22:25.712 --> 00:22:26.632
<00:22:25.712>since <00:22:26.126>add <00:22:26.540>edge

370
00:22:27.452 --> 00:22:31.272
<00:22:27.452>does <00:22:27.627>several <00:22:27.802>writes <00:22:27.977>per <00:22:28.152>call. <00:22:28.327>The <00:22:28.502>conflict <00:22:28.677>policy <00:22:28.852>is

371
00:22:31.392 --> 00:22:35.232
<00:22:31.392>data <00:22:31.547>and <00:22:31.703>logic <00:22:31.858>in <00:22:32.014>one <00:22:32.169>place, <00:22:32.325>not <00:22:32.480>smeared <00:22:32.636>across <00:22:32.792>application

372
00:22:35.332 --> 00:22:35.712
code.

373
00:22:39.512 --> 00:22:43.212
<00:22:39.512>Two <00:22:39.712>query <00:22:39.912>shapes. <00:22:40.111>A <00:22:40.312>snapshot <00:22:40.512>of <00:22:40.712>all <00:22:40.912>active

374
00:22:43.312 --> 00:22:46.832
<00:22:43.312>couplings <00:22:43.512>at <00:22:43.712>time <00:22:43.911>T <00:22:44.111>is <00:22:44.312>just <00:22:44.512>an <00:22:44.712>interval

375
00:22:46.872 --> 00:22:49.892
<00:22:46.872>predicate. <00:22:47.105>Temporal <00:22:47.338>querying <00:22:47.572>is <00:22:47.805>a <00:22:48.038>rare <00:22:48.272>clause.

376
00:22:50.992 --> 00:22:54.432
<00:22:50.992>The <00:22:51.225>second <00:22:51.458>reconstructs <00:22:51.692>a <00:22:51.925>whole <00:22:52.158>road <00:22:52.392>train

377
00:22:54.692 --> 00:22:58.612
<00:22:54.692>below <00:22:54.892>a <00:22:55.092>vehicle <00:22:55.292>at <00:22:55.492>an <00:22:55.692>instant, <00:22:55.892>recursively <00:22:56.092>following

378
00:22:58.692 --> 00:23:01.152
<00:22:58.692>edges <00:22:59.042>whose <00:22:59.392>intervals <00:22:59.742>contain <00:23:00.092>T.

379
00:23:02.132 --> 00:23:05.782
<00:23:02.132>The <00:23:02.365>GraphDB <00:23:02.598>use <00:23:02.832>case, <00:23:03.065>transitive <00:23:03.298>reachability <00:23:03.532>and

380
00:23:05.912 --> 00:23:07.992
<00:23:05.912>standard <00:23:06.612>recursive <00:23:07.312>SQL.

381
00:23:10.152 --> 00:23:14.092
<00:23:10.152>And <00:23:10.279>you <00:23:10.406>can <00:23:10.533>ask <00:23:10.661>this <00:23:10.788>for <00:23:10.915>any <00:23:11.042>historical <00:23:11.170>T, <00:23:11.297>because <00:23:11.424>we <00:23:11.552>never

382
00:23:14.152 --> 00:23:17.292
<00:23:14.152>overwrite <00:23:14.502>history. <00:23:14.852>We <00:23:15.202>adjust <00:23:15.552>intervals.

383
00:23:17.952 --> 00:23:21.652
<00:23:17.952>Full <00:23:18.652>temporal <00:23:19.352>auditability.

384
00:23:24.272 --> 00:23:25.672
<00:23:24.272>So <00:23:24.692>the <00:23:25.112>third <00:23:25.532>takeaway,

385
00:23:26.552 --> 00:23:29.612
<00:23:26.552>three <00:23:26.752>problems <00:23:26.952>people <00:23:27.151>reach <00:23:27.351>for <00:23:27.552>several <00:23:27.752>tooling <00:23:27.952>for.

386
00:23:30.112 --> 00:23:33.712
<00:23:30.112>GraphDB, <00:23:30.345>temporal <00:23:30.578>framework, <00:23:30.812>app <00:23:31.045>layer <00:23:31.278>conflict <00:23:31.512>logic

387
00:23:34.032 --> 00:23:36.212
<00:23:34.032>collapse <00:23:34.381>into <00:23:34.732>one <00:23:35.081>well-constrained <00:23:35.432>table.

388
00:23:37.052 --> 00:23:40.442
<00:23:37.052>The <00:23:37.252>strongest <00:23:37.452>argument <00:23:37.651>here <00:23:37.851>isn't <00:23:38.052>fewer <00:23:38.252>services, <00:23:38.452>it's

389
00:23:40.472 --> 00:23:43.932
<00:23:40.472>correctness, <00:23:40.822>as <00:23:41.172>impossible <00:23:41.522>states <00:23:41.872>are

390
00:23:44.012 --> 00:23:47.292
<00:23:44.012>unrepresentable <00:23:44.292>enforced <00:23:44.572>at <00:23:44.851>the <00:23:45.131>storage <00:23:45.412>layer.

391
00:23:48.352 --> 00:23:51.732
<00:23:48.352>This <00:23:48.527>part <00:23:48.702>is <00:23:48.877>also <00:23:49.052>open <00:23:49.227>sourced. <00:23:49.402>Have <00:23:49.577>a <00:23:49.752>look.

392
00:23:54.732 --> 00:23:58.512
<00:23:54.732>On <00:23:54.859>to <00:23:54.986>the <00:23:55.113>last <00:23:55.241>deep <00:23:55.368>dive, <00:23:55.495>the <00:23:55.622>life <00:23:55.750>cycle <00:23:55.877>problem <00:23:56.004>of <00:23:56.132>any

393
00:23:58.552 --> 00:24:02.092
<00:23:58.552>telemetry <00:23:58.752>system. <00:23:58.952>Volume <00:23:59.151>only <00:23:59.351>grows, <00:23:59.552>but <00:23:59.752>access <00:23:59.952>is

394
00:24:02.152 --> 00:24:05.852
<00:24:02.152>heavily <00:24:02.352>skewed <00:24:02.552>to <00:24:02.752>recent <00:24:02.952>data. <00:24:03.152>Keeping <00:24:03.352>years <00:24:03.552>of

395
00:24:05.892 --> 00:24:09.632
<00:24:05.892>GPS <00:24:06.047>in <00:24:06.203>your <00:24:06.358>primary <00:24:06.514>database <00:24:06.669>is <00:24:06.825>expensive <00:24:06.980>and <00:24:07.136>slows <00:24:07.292>everything

396
00:24:09.672 --> 00:24:12.472
<00:24:09.672>down. <00:24:09.952>And <00:24:10.232>remember <00:24:10.512>those <00:24:10.792>old <00:24:11.072>partitions?

397
00:24:12.752 --> 00:24:13.842
<00:24:12.752>They <00:24:12.997>are <00:24:13.242>the <00:24:13.487>natural <00:24:13.733>unit

398
00:24:15.592 --> 00:24:18.952
<00:24:15.592>The <00:24:15.732>conventional <00:24:15.872>fix <00:24:16.012>is <00:24:16.152>a <00:24:16.292>data <00:24:16.432>lake <00:24:16.572>plus <00:24:16.712>a <00:24:16.852>query <00:24:16.992>layer.

399
00:24:20.012 --> 00:24:23.572
<00:24:20.012>A <00:24:20.167>whole <00:24:20.323>second <00:24:20.478>data <00:24:20.634>platform <00:24:20.789>to <00:24:20.945>build, <00:24:21.100>secure, <00:24:21.256>and <00:24:21.412>operate.

400
00:24:23.932 --> 00:24:26.252
<00:24:23.932>Exactly <00:24:24.132>the <00:24:24.332>moving <00:24:24.532>part <00:24:24.732>we <00:24:24.932>want <00:24:25.132>to <00:24:25.332>avoid.

401
00:24:26.692 --> 00:24:30.652
<00:24:26.692>Our <00:24:26.832>alternative, <00:24:26.972>cheap <00:24:27.112>object <00:24:27.252>storage <00:24:27.392>and <00:24:27.532>push <00:24:27.672>the <00:24:27.812>query <00:24:27.952>to <00:24:28.092>the

402
00:24:30.712 --> 00:24:31.192
client.

403
00:24:33.612 --> 00:24:37.202
<00:24:33.612>So <00:24:33.739>our <00:24:33.866>pipeline, <00:24:33.993>when <00:24:34.121>a <00:24:34.248>partition <00:24:34.375>ages <00:24:34.502>out, <00:24:34.630>we <00:24:34.757>export <00:24:34.884>it <00:24:35.012>to

404
00:24:37.312 --> 00:24:40.122
<00:24:37.312>Parquet <00:24:37.545>and <00:24:37.778>drop <00:24:38.012>it <00:24:38.245>in <00:24:38.478>S3. <00:24:38.712>Columnar,

405
00:24:40.832 --> 00:24:42.492
<00:24:40.832>compressed, <00:24:41.298>an <00:24:41.765>open <00:24:42.232>format,

406
00:24:42.712 --> 00:24:46.412
<00:24:42.712>ideal <00:24:43.412>for <00:24:44.112>scan-heavy

407
00:24:46.452 --> 00:24:47.672
<00:24:46.452>analytical <00:24:47.550>queries.

408
00:24:48.632 --> 00:24:51.372
<00:24:48.632>The <00:24:48.865>surprising <00:24:49.098>part, <00:24:49.332>the <00:24:49.565>browser <00:24:49.798>queries <00:24:50.032>it.

409
00:24:51.832 --> 00:24:55.532
<00:24:51.832>DuckDB-Wasm <00:24:52.182>is <00:24:52.532>a <00:24:52.882>full <00:24:53.232>analytical

410
00:24:55.712 --> 00:24:58.572
<00:24:55.712>SQL <00:24:56.062>engine <00:24:56.412>compiled <00:24:56.762>to <00:24:57.112>WebAssembly.

411
00:24:58.652 --> 00:25:02.112
<00:24:58.652>It <00:24:58.852>fetches <00:24:59.052>a <00:24:59.252>Parquet <00:24:59.452>file <00:24:59.652>over <00:24:59.852>a <00:25:00.052>signed

412
00:25:02.252 --> 00:25:06.032
<00:25:02.252>URL, <00:25:02.532>even <00:25:02.812>range <00:25:03.091>requests, <00:25:03.371>just <00:25:03.652>the

413
00:25:06.092 --> 00:25:08.872
<00:25:06.092>columns <00:25:06.325>it <00:25:06.558>needs, <00:25:06.792>and <00:25:07.025>runs <00:25:07.258>SQL <00:25:07.492>locally.

414
00:25:09.512 --> 00:25:13.152
<00:25:09.512>So <00:25:09.712>there's <00:25:09.912>no <00:25:10.111>server-side <00:25:10.312>query <00:25:10.512>service <00:25:10.712>for <00:25:10.912>cold

415
00:25:13.192 --> 00:25:17.132
<00:25:13.192>data <00:25:13.425>at <00:25:13.658>all. <00:25:13.892>Access <00:25:14.125>is <00:25:14.358>via <00:25:14.592>per-customer

416
00:25:17.232 --> 00:25:20.652
<00:25:17.232>signed <00:25:17.387>URLs, <00:25:17.543>and <00:25:17.698>the <00:25:17.854>data <00:25:18.009>never <00:25:18.165>passes <00:25:18.320>through <00:25:18.476>an <00:25:18.632>app

417
00:25:21.492 --> 00:25:21.832
server.

418
00:25:24.792 --> 00:25:28.192
<00:25:24.792>S3 <00:25:24.966>is <00:25:25.141>about <00:25:25.317>as <00:25:25.492>close <00:25:25.667>to <00:25:25.841>zero <00:25:26.016>maintenance <00:25:26.192>as

419
00:25:28.252 --> 00:25:31.992
<00:25:28.252>infrastructure <00:25:28.485>gets, <00:25:28.718>and <00:25:28.952>Parquet <00:25:29.185>being <00:25:29.418>open <00:25:29.652>means

420
00:25:32.092 --> 00:25:33.612
<00:25:32.092>even <00:25:32.365>this <00:25:32.639>layer <00:25:32.912>is <00:25:33.186>in <00:25:33.460>login.

421
00:25:35.632 --> 00:25:39.472
<00:25:35.632>Anything <00:25:35.787>can <00:25:35.943>read <00:25:36.098>those <00:25:36.254>files <00:25:36.409>later. <00:25:36.565>The <00:25:36.720>cost <00:25:36.876>model <00:25:37.032>flips.

422
00:25:40.392 --> 00:25:44.172
<00:25:40.392>A <00:25:40.547>heavy <00:25:40.703>query <00:25:40.858>over <00:25:41.014>years <00:25:41.169>of <00:25:41.325>data <00:25:41.480>costs <00:25:41.636>us <00:25:41.792>nothing

423
00:25:44.252 --> 00:25:47.892
<00:25:44.252>server <00:25:44.392>side <00:25:44.532>because <00:25:44.672>it <00:25:44.812>runs <00:25:44.952>on <00:25:45.091>the <00:25:45.232>user's <00:25:45.371>browser, <00:25:45.512>so <00:25:45.652>it

424
00:25:47.992 --> 00:25:51.582
<00:25:47.992>scales <00:25:48.167>for <00:25:48.341>free <00:25:48.517>with <00:25:48.692>users, <00:25:48.867>and <00:25:49.042>it's <00:25:49.216>SQL <00:25:49.392>and

425
00:25:51.612 --> 00:25:55.592
<00:25:51.612>Postgres, <00:25:51.812>and <00:25:52.012>SQL <00:25:52.212>and <00:25:52.412>DuckDB. <00:25:52.612>Nobody <00:25:52.812>learns <00:25:53.012>a

426
00:25:55.792 --> 00:25:56.812
<00:25:55.792>bespoke <00:25:56.709>language.

427
00:25:58.692 --> 00:26:00.632
<00:25:58.692>One <00:25:59.392>unescaped <00:26:00.092>caveat.

428
00:26:01.872 --> 00:26:05.572
<00:26:01.872>This <00:26:02.072>fits <00:26:02.272>analytical <00:26:02.472>read-mostly <00:26:02.672>cold <00:26:02.872>access, <00:26:03.072>not <00:26:03.272>low

429
00:26:05.632 --> 00:26:09.152
<00:26:05.632>latency <00:26:05.865>point <00:26:06.098>lookups, <00:26:06.332>and <00:26:06.565>very <00:26:06.798>large <00:26:07.032>scans

430
00:26:09.312 --> 00:26:10.972
<00:26:09.312>still <00:26:09.591>pull <00:26:09.871>bytes <00:26:10.151>to <00:26:10.431>the <00:26:10.712>client.

431
00:26:11.832 --> 00:26:13.142
<00:26:11.832>Right <00:26:12.126>tool <00:26:12.421>for <00:26:12.716>the <00:26:13.011>cold

432
00:26:14.352 --> 00:26:15.632
<00:26:14.352>tier <00:26:15.504>specifically.

433
00:26:19.172 --> 00:26:22.752
<00:26:19.172>So <00:26:19.327>we <00:26:19.483>deleted <00:26:19.638>the <00:26:19.794>data <00:26:19.949>lake <00:26:20.105>query <00:26:20.260>service <00:26:20.416>from <00:26:20.572>the

434
00:26:22.792 --> 00:26:26.692
<00:26:22.792>diagram <00:26:23.025>entirely, <00:26:23.258>and <00:26:23.492>that's <00:26:23.725>the <00:26:23.958>through <00:26:24.192>line

435
00:26:26.752 --> 00:26:28.232
<00:26:26.752>of <00:26:27.085>all <00:26:27.418>four <00:26:27.751>deep <00:26:28.084>dives.

436
00:26:29.112 --> 00:26:31.532
<00:26:29.112>Each <00:26:29.345>one <00:26:29.578>removed <00:26:29.812>a <00:26:30.045>category <00:26:30.278>of <00:26:30.512>service.

437
00:26:31.752 --> 00:26:35.072
<00:26:31.752>A <00:26:31.952>stream <00:26:32.152>processor <00:26:32.351>drives <00:26:32.552>GraphDB <00:26:32.752>data <00:26:32.952>lake <00:26:33.152>engine

438
00:26:35.652 --> 00:26:37.952
<00:26:35.652>by <00:26:35.885>leaning <00:26:36.118>on <00:26:36.352>Postgres <00:26:36.585>and <00:26:36.818>open <00:26:37.052>formats.

439
00:26:38.512 --> 00:26:40.052
<00:26:38.512>Now, <00:26:38.974>let's <00:26:39.436>zoom <00:26:39.898>out.

440
00:26:41.652 --> 00:26:45.492
<00:26:41.652>We've <00:26:41.807>gone <00:26:41.963>deep <00:26:42.118>a <00:26:42.274>few <00:26:42.429>times <00:26:42.585>and <00:26:42.740>now <00:26:42.896>reassemble <00:26:43.052>the

441
00:26:45.552 --> 00:26:49.492
<00:26:45.552>whole <00:26:45.679>stack <00:26:45.806>in <00:26:45.933>one <00:26:46.061>view <00:26:46.188>to <00:26:46.315>see <00:26:46.442>how <00:26:46.570>few <00:26:46.697>parts <00:26:46.824>there <00:26:46.952>really

442
00:26:49.572 --> 00:26:49.792
are.

443
00:26:51.872 --> 00:26:55.572
<00:26:51.872>From <00:26:52.047>top <00:26:52.222>to <00:26:52.397>bottom, <00:26:52.572>the <00:26:52.747>four <00:26:52.922>infrastructure <00:26:53.097>tools <00:26:53.272>hold

444
00:26:55.672 --> 00:26:59.232
<00:26:55.672>everything <00:26:55.905>up. <00:26:56.138>Ingest, <00:26:56.372>then <00:26:56.605>go <00:26:56.838>to <00:26:57.072>the

445
00:26:59.352 --> 00:27:00.132
<00:26:59.352>actual <00:27:00.054>system.

446
00:27:00.952 --> 00:27:03.912
<00:27:00.952>Ingest <00:27:01.152>lands <00:27:01.352>in <00:27:01.552>Postgres <00:27:01.752>and <00:27:01.952>PostGIS <00:27:02.152>on <00:27:02.352>Neon.

447
00:27:04.172 --> 00:27:08.052
<00:27:04.172>The <00:27:04.288>single <00:27:04.405>source <00:27:04.522>of <00:27:04.638>truth <00:27:04.755>where <00:27:04.872>the <00:27:04.988>Kalman <00:27:05.105>filter <00:27:05.222>and <00:27:05.338>the <00:27:05.455>temporal <00:27:05.572>graphs

448
00:27:08.132 --> 00:27:11.492
<00:27:08.132>live. <00:27:08.365>Telemetry <00:27:08.598>is <00:27:08.832>partitioned <00:27:09.065>and <00:27:09.298>age <00:27:09.532>partitions

449
00:27:12.692 --> 00:27:15.932
<00:27:12.692>become <00:27:12.892>Parquet <00:27:13.092>and <00:27:13.292>S3, <00:27:13.492>with <00:27:13.692>the <00:27:13.892>browser <00:27:14.092>queries

450
00:27:16.052 --> 00:27:18.212
<00:27:16.052>directly <00:27:16.752>with <00:27:17.452>DuckDB-Wasm.

451
00:27:20.532 --> 00:27:22.472
<00:27:20.532>Count <00:27:20.812>the <00:27:21.091>parts <00:27:21.371>we <00:27:21.651>actually <00:27:21.932>operate.

452
00:27:23.512 --> 00:27:27.332
<00:27:23.512>Postgres <00:27:23.667>managed <00:27:23.823>by <00:27:23.978>Neon, <00:27:24.134>handful <00:27:24.289>of <00:27:24.445>Nomad <00:27:24.600>jobs, <00:27:24.756>and <00:27:24.912>an

453
00:27:27.492 --> 00:27:29.592
<00:27:27.492>S3 <00:27:27.841>bucket. <00:27:28.192>That's <00:27:28.542>essentially <00:27:28.892>it.

454
00:27:30.612 --> 00:27:33.712
<00:27:30.612>Compared <00:27:30.787>to <00:27:30.962>the <00:27:31.137>complexity <00:27:31.312>magnet <00:27:31.487>slide <00:27:31.662>from <00:27:31.837>the <00:27:32.012>start,

455
00:27:34.472 --> 00:27:36.972
<00:27:34.472>every <00:27:34.672>place <00:27:34.872>we <00:27:35.072>could <00:27:35.272>have <00:27:35.472>added <00:27:35.672>a <00:27:35.872>service,

456
00:27:37.812 --> 00:27:41.692
<00:27:37.812>we <00:27:37.928>first <00:27:38.045>asked <00:27:38.161>whether <00:27:38.278>the <00:27:38.395>database <00:27:38.512>or <00:27:38.628>an <00:27:38.745>open <00:27:38.861>format <00:27:38.978>could <00:27:39.095>do <00:27:39.212>the

457
00:27:41.752 --> 00:27:45.592
<00:27:41.752>job. <00:27:41.952>And <00:27:42.152>surprisingly <00:27:42.351>often, <00:27:42.552>the <00:27:42.752>answer <00:27:42.952>is <00:27:43.152>yes.

458
00:27:49.452 --> 00:27:50.752
<00:27:49.452>Of <00:27:49.842>course, <00:27:50.232>there's <00:27:50.621>trade-offs,

459
00:27:52.312 --> 00:27:54.592
<00:27:52.312>so <00:27:52.545>I <00:27:52.778>cannot <00:27:53.012>only <00:27:53.245>sell <00:27:53.478>the <00:27:53.712>upside.

460
00:27:57.672 --> 00:27:58.232
<00:27:57.672>First <00:27:57.924>of <00:27:58.176>all,

461
00:27:59.372 --> 00:28:02.312
<00:27:59.372>PLPG <00:27:59.652>SQL <00:27:59.932>as <00:28:00.212>a <00:28:00.492>language. <00:28:00.772>So

462
00:28:03.192 --> 00:28:04.252
<00:28:03.192>few <00:28:03.510>people <00:28:03.828>know <00:28:04.146>it,

463
00:28:05.072 --> 00:28:08.572
<00:28:05.072>so <00:28:05.305>you <00:28:05.538>have <00:28:05.772>to <00:28:06.005>invest <00:28:06.238>a <00:28:06.472>bit.

464
00:28:09.492 --> 00:28:13.232
<00:28:09.492>The <00:28:09.958>in-database <00:28:10.425>computation, <00:28:10.892>of

465
00:28:13.312 --> 00:28:16.452
<00:28:13.312>course, <00:28:13.661>competes <00:28:14.012>with <00:28:14.361>the <00:28:14.712>ongoing

466
00:28:16.592 --> 00:28:17.352
OLTP.

467
00:28:18.972 --> 00:28:22.312
<00:28:18.972>That <00:28:19.147>is <00:28:19.322>a <00:28:19.497>problem. <00:28:19.672>We <00:28:19.847>try <00:28:20.022>to <00:28:20.197>mitigate <00:28:20.372>this

468
00:28:22.832 --> 00:28:26.792
<00:28:22.832>using <00:28:23.182>multiple <00:28:23.532>computes <00:28:23.882>in <00:28:24.232>Neon.

469
00:28:27.532 --> 00:28:29.232
<00:28:27.532>That <00:28:27.998>is <00:28:28.465>possible <00:28:28.932>because

470
00:28:30.612 --> 00:28:33.051
<00:28:30.612>storage <00:28:30.962>and <00:28:31.312>computes <00:28:31.662>are <00:28:32.012>separated.

471
00:28:41.252 --> 00:28:45.232
<00:28:41.252>We <00:28:41.601>lean <00:28:41.952>on <00:28:42.302>Postgres-specific <00:28:42.652>features,

472
00:28:46.132 --> 00:28:46.952
<00:28:46.132>which <00:28:46.501>is <00:28:46.870>coupling,

473
00:28:47.952 --> 00:28:51.032
<00:28:47.952>but <00:28:48.107>the <00:28:48.263>data <00:28:48.418>stays <00:28:48.574>in <00:28:48.729>open <00:28:48.885>formats, <00:28:49.040>SQL <00:28:49.196>and <00:28:49.352>Parquet.

474
00:28:51.472 --> 00:28:51.552
So

475
00:28:53.132 --> 00:28:56.992
<00:28:53.132>exit <00:28:53.272>is <00:28:53.412>a <00:28:53.552>migration, <00:28:53.692>not <00:28:53.832>a <00:28:53.972>rewrite, <00:28:54.112>and <00:28:54.252>clever <00:28:54.392>SQL <00:28:54.532>is

476
00:28:57.032 --> 00:28:59.232
<00:28:57.032>still <00:28:57.381>clever <00:28:57.732>code. <00:28:58.081>Comment <00:28:58.432>it.

477
00:29:00.892 --> 00:29:04.052
<00:29:00.892>Honest <00:29:01.125>conclusion, <00:29:01.358>this <00:29:01.592>is <00:29:01.825>a <00:29:02.058>scale-dependent <00:29:02.292>trait.

478
00:29:04.462 --> 00:29:06.612
<00:29:04.462>At <00:29:04.742>much <00:29:05.022>larger <00:29:05.302>scale, <00:29:05.581>different <00:29:05.862>answer.

479
00:29:07.012 --> 00:29:10.312
<00:29:07.012>For <00:29:07.167>a <00:29:07.323>small <00:29:07.478>team <00:29:07.634>running <00:29:07.789>a <00:29:07.945>real <00:29:08.100>fleet, <00:29:08.256>simplicity <00:29:08.412>wins.

480
00:29:14.672 --> 00:29:18.532
<00:29:14.672>So <00:29:14.905>here's <00:29:15.138>the <00:29:15.372>take-home <00:29:15.605>lessons. <00:29:15.838>Five <00:29:16.072>portable

481
00:29:18.592 --> 00:29:22.072
<00:29:18.592>lessons. <00:29:18.792>None <00:29:18.992>of <00:29:19.192>that <00:29:19.392>is <00:29:19.592>specific <00:29:19.792>to <00:29:19.992>fleets.

482
00:29:23.132 --> 00:29:25.232
<00:29:23.132>One, <00:29:23.482>count <00:29:23.832>your <00:29:24.182>moving <00:29:24.532>parts.

483
00:29:26.372 --> 00:29:27.712
<00:29:26.372>Most <00:29:26.774>of <00:29:27.176>a <00:29:27.578>component's

484
00:29:28.672 --> 00:29:30.672
<00:29:28.672>cost <00:29:28.952>comes <00:29:29.232>after <00:29:29.512>you <00:29:29.792>ship <00:29:30.072>it.

485
00:29:32.432 --> 00:29:33.232
<00:29:32.432>Takeaway <00:29:33.152>two,

486
00:29:34.652 --> 00:29:38.632
<00:29:34.652>push <00:29:34.779>logic <00:29:34.906>to <00:29:35.033>the <00:29:35.161>data <00:29:35.288>before <00:29:35.415>you <00:29:35.542>push <00:29:35.670>data <00:29:35.797>to <00:29:35.924>new <00:29:36.052>services.

487
00:29:39.392 --> 00:29:43.352
<00:29:39.392>Three, <00:29:39.532>one <00:29:39.672>source <00:29:39.812>of <00:29:39.952>truth <00:29:40.092>beats <00:29:40.232>N <00:29:40.372>stores <00:29:40.512>you <00:29:40.652>keep <00:29:40.792>in

488
00:29:43.392 --> 00:29:43.672
sync.

489
00:29:45.252 --> 00:29:48.952
<00:29:45.252>Four, <00:29:45.392>and <00:29:45.532>this <00:29:45.672>ties <00:29:45.812>to <00:29:45.952>the <00:29:46.091>cloud <00:29:46.232>point <00:29:46.371>from <00:29:46.512>earlier, <00:29:46.652>prefer

490
00:29:49.092 --> 00:29:52.422
<00:29:49.092>one <00:29:49.232>tool, <00:29:49.372>one <00:29:49.512>job, <00:29:49.652>and <00:29:49.792>keep <00:29:49.932>open <00:29:50.072>formats <00:29:50.212>at <00:29:50.352>the <00:29:50.492>boundaries,

491
00:29:52.892 --> 00:29:56.492
<00:29:52.892>because <00:29:53.047>that's <00:29:53.203>what <00:29:53.358>keeps <00:29:53.514>the <00:29:53.669>exit <00:29:53.825>cheap <00:29:53.980>when <00:29:54.136>a <00:29:54.292>cloud

492
00:29:56.532 --> 00:29:58.852
<00:29:56.532>service <00:29:56.765>bill <00:29:56.998>or <00:29:57.232>lock-in <00:29:57.465>turns <00:29:57.698>against <00:29:57.932>you.

493
00:29:59.352 --> 00:30:02.952
<00:29:59.352>And <00:29:59.527>five, <00:29:59.702>engineer <00:29:59.877>your <00:30:00.052>actual <00:30:00.227>scale, <00:30:00.402>not <00:30:00.577>the <00:30:00.752>scale

494
00:30:03.492 --> 00:30:04.712
<00:30:03.492>on <00:30:03.858>the <00:30:04.224>conference <00:30:04.590>slides.

495
00:30:08.452 --> 00:30:09.372
<00:30:08.452>So <00:30:08.866>to <00:30:09.280>close,

496
00:30:10.532 --> 00:30:14.272
<00:30:10.532>our <00:30:10.687>simplicity <00:30:10.843>came <00:30:10.998>from <00:30:11.154>keeping <00:30:11.309>Postgres <00:30:11.465>at <00:30:11.620>the <00:30:11.776>center <00:30:11.932>and

497
00:30:14.372 --> 00:30:17.572
<00:30:14.372>only <00:30:14.605>adding <00:30:14.838>moving <00:30:15.072>parts <00:30:15.305>when <00:30:15.538>the <00:30:15.772>database

498
00:30:17.672 --> 00:30:19.252
genuinely

499
00:30:20.832 --> 00:30:24.352
<00:30:20.832>really <00:30:20.972>couldn't <00:30:21.112>do <00:30:21.252>the <00:30:21.392>job. <00:30:21.532>The <00:30:21.672>two <00:30:21.812>open <00:30:21.952>repos <00:30:22.092>are <00:30:22.232>both

500
00:30:24.452 --> 00:30:27.532
<00:30:24.452>runnable <00:30:24.607>on <00:30:24.763>a <00:30:24.918>Neon <00:30:25.074>branch. <00:30:25.229>Scan <00:30:25.385>the <00:30:25.540>codes <00:30:25.696>to <00:30:25.852>explore.

501
00:30:28.072 --> 00:30:29.352
<00:30:28.072>Thank <00:30:28.456>you <00:30:28.840>for <00:30:29.224>watching.