WEBVTT

1
00:00:02.575 --> 00:00:03.195
<00:00:02.575>Hi, <00:00:03.133>everybody.

2
00:00:03.405 --> 00:00:06.835
<00:00:03.405>Hello. <00:00:03.579>Uh, <00:00:03.755>let's, <00:00:03.929>uh, <00:00:04.104>start <00:00:04.279>off <00:00:04.455>with <00:00:04.630>some <00:00:04.805>introductions.

3
00:00:07.335 --> 00:00:08.355
<00:00:07.335>So <00:00:07.794>I'm <00:00:08.253>Oli.

4
00:00:08.655 --> 00:00:11.115
<00:00:08.655>Uh, <00:00:08.795>my <00:00:08.934>speaker <00:00:09.075>profile <00:00:09.215>is <00:00:09.354>a <00:00:09.495>little <00:00:09.635>bit <00:00:09.774>out <00:00:09.915>of <00:00:10.055>date.

5
00:00:11.255 --> 00:00:13.795
<00:00:11.255>I'm <00:00:11.455>actually <00:00:11.655>an <00:00:11.855>engineering <00:00:12.055>manager <00:00:12.255>at, <00:00:12.455>uh, <00:00:12.655>Teleo,

6
00:00:14.135 --> 00:00:17.875
<00:00:14.135>but <00:00:14.251>I'm <00:00:14.368>here <00:00:14.485>today <00:00:14.601>to <00:00:14.718>talk <00:00:14.834>to <00:00:14.951>you <00:00:15.068>about <00:00:15.184>the <00:00:15.301>work <00:00:15.418>I <00:00:15.535>did

7
00:00:17.875 --> 00:00:21.515
<00:00:17.875>with <00:00:18.075>Dennis, <00:00:18.275>uh, <00:00:18.475>for <00:00:18.675>my <00:00:18.875>previous <00:00:19.075>employer, <00:00:19.275>futurist.

8
00:00:22.175 --> 00:00:24.955
<00:00:22.175>Hey, <00:00:22.330>and <00:00:22.486>I'm <00:00:22.641>Dennis. <00:00:22.797>I'm <00:00:22.952>still <00:00:23.108>a <00:00:23.263>senior <00:00:23.419>developer <00:00:23.575>at

9
00:00:25.155 --> 00:00:26.675
<00:00:25.155>Futurist, <00:00:25.497>and <00:00:25.839>I'm <00:00:26.181>currently <00:00:26.523>working

10
00:00:26.695 --> 00:00:28.955
<00:00:26.695>as <00:00:26.850>a <00:00:27.006>tech <00:00:27.161>lead <00:00:27.317>on <00:00:27.472>the <00:00:27.628>system <00:00:27.783>that <00:00:27.939>we're <00:00:28.095>about

11
00:00:28.955 --> 00:00:30.035
<00:00:28.955>to <00:00:29.149>show <00:00:29.343>you <00:00:29.538>in <00:00:29.732>this <00:00:29.927>talk.

12
00:00:30.895 --> 00:00:33.275
<00:00:30.895>So <00:00:31.035>we're <00:00:31.175>going <00:00:31.314>to <00:00:31.455>talk <00:00:31.595>about <00:00:31.735>this <00:00:31.875>project <00:00:32.015>that <00:00:32.155>we <00:00:32.295>did

13
00:00:33.295 --> 00:00:34.715
<00:00:33.295>for <00:00:33.614>a <00:00:33.934>client <00:00:34.253>of <00:00:34.573>ours,

14
00:00:35.215 --> 00:00:38.515
<00:00:35.215>and, <00:00:35.390>uh, <00:00:35.565>it <00:00:35.740>was <00:00:35.915>basically <00:00:36.090>building <00:00:36.265>a, <00:00:36.440>a <00:00:36.615>platform

15
00:00:39.015 --> 00:00:41.915
<00:00:39.015>of <00:00:39.295>microservices <00:00:39.575>that <00:00:39.855>running <00:00:40.135>Kafka <00:00:40.415>streams.

16
00:00:42.375 --> 00:00:46.795
<00:00:42.375>So <00:00:42.530>the <00:00:42.686>client <00:00:42.841>was <00:00:42.997>a <00:00:43.152>major, <00:00:43.308>uh, <00:00:43.463>global <00:00:43.619>German <00:00:43.775>brand,

17
00:00:47.175 --> 00:00:50.435
<00:00:47.175>and <00:00:47.302>I <00:00:47.429>would, <00:00:47.556>uh, <00:00:47.684>suspect <00:00:47.811>that <00:00:47.938>99% <00:00:48.065>of <00:00:48.193>you <00:00:48.320>have <00:00:48.447>either <00:00:48.574>heard

18
00:00:50.435 --> 00:00:52.315
<00:00:50.435>of <00:00:50.635>them <00:00:50.835>or <00:00:51.035>maybe <00:00:51.235>even <00:00:51.435>used <00:00:51.635>their <00:00:51.835>products.

19
00:00:53.135 --> 00:00:57.875
<00:00:53.135>Um, <00:00:53.262>so <00:00:53.389>this <00:00:53.516>project <00:00:53.644>that, <00:00:53.771>uh, <00:00:53.898>started <00:00:54.025>in <00:00:54.153>the <00:00:54.280>summer <00:00:54.407>of <00:00:54.535>2018

20
00:00:58.655 --> 00:01:02.275
<00:00:58.655>and <00:00:58.782>as <00:00:58.909>an <00:00:59.036>ambitious <00:00:59.164>goal, <00:00:59.291>uh, <00:00:59.418>with <00:00:59.545>the <00:00:59.673>client <00:00:59.800>at <00:00:59.927>the <00:01:00.055>time,

21
00:01:02.815 --> 00:01:05.155
<00:01:02.815>uh, <00:01:02.970>we <00:01:03.126>decided <00:01:03.281>that <00:01:03.437>we <00:01:03.592>would <00:01:03.748>like <00:01:03.903>to <00:01:04.059>do <00:01:04.215>something

22
00:01:05.215 --> 00:01:08.115
<00:01:05.215>so <00:01:05.415>interesting <00:01:05.615>that <00:01:05.815>we <00:01:06.015>could <00:01:06.215>make <00:01:06.415>a <00:01:06.615>compelling

23
00:01:08.115 --> 00:01:09.435
<00:01:08.115>conference <00:01:08.411>talk <00:01:08.709>out <00:01:09.006>of <00:01:09.303>it.

24
00:01:09.695 --> 00:01:12.275
<00:01:09.695>So <00:01:09.869>here <00:01:10.044>we <00:01:10.220>are. <00:01:10.395>Let's <00:01:10.570>hope <00:01:10.744>it's <00:01:10.919>interesting <00:01:11.095>material.

25
00:01:13.535 --> 00:01:16.115
<00:01:13.535>All <00:01:13.675>right, <00:01:13.815>we'll <00:01:13.955>start <00:01:14.095>off <00:01:14.235>with <00:01:14.375>a <00:01:14.515>little <00:01:14.655>bit <00:01:14.795>of <00:01:14.935>introduction

26
00:01:16.115 --> 00:01:18.275
<00:01:16.115>to <00:01:16.289>Kafka <00:01:16.464>and <00:01:16.640>Kafka <00:01:16.815>streams, <00:01:16.990>just <00:01:17.164>to <00:01:17.339>make <00:01:17.515>sure

27
00:01:18.275 --> 00:01:19.715
<00:01:18.275>that <00:01:18.534>we're <00:01:18.793>all <00:01:19.052>using <00:01:19.311>the <00:01:19.571>same

28
00:01:19.725 --> 00:01:21.315
<00:01:19.725>vocabulary <00:01:20.074>and <00:01:20.425>understand <00:01:20.774>each <00:01:21.125>other.

29
00:01:21.935 --> 00:01:24.515
<00:01:21.935>And <00:01:22.135>then <00:01:22.335>we'll <00:01:22.535>talk <00:01:22.735>about, <00:01:22.935>uh, <00:01:23.135>our <00:01:23.335>journey,

30
00:01:24.975 --> 00:01:28.155
<00:01:24.975>how <00:01:25.130>we <00:01:25.286>actually <00:01:25.441>evolved <00:01:25.597>to <00:01:25.752>the <00:01:25.908>design <00:01:26.063>of <00:01:26.219>Kafka <00:01:26.375>Streams,

31
00:01:28.155 --> 00:01:30.515
<00:01:28.155>microservices, <00:01:28.435>starting <00:01:28.715>from <00:01:28.995>the <00:01:29.275>client <00:01:29.555>problem.

32
00:01:31.135 --> 00:01:33.315
<00:01:31.135>And <00:01:31.335>then <00:01:31.535>we'll <00:01:31.735>talk <00:01:31.935>about <00:01:32.135>real <00:01:32.335>life <00:01:32.535>examples

33
00:01:33.315 --> 00:01:34.755
<00:01:33.315>of <00:01:33.574>data <00:01:33.833>pipelines <00:01:34.092>that <00:01:34.351>we <00:01:34.610>built

34
00:01:35.055 --> 00:01:37.235
<00:01:35.055>and <00:01:35.230>the <00:01:35.405>lessons <00:01:35.580>that <00:01:35.755>we <00:01:35.930>learned <00:01:36.105>along <00:01:36.280>the <00:01:36.455>way,

35
00:01:37.245 --> 00:01:38.395
<00:01:37.245>which <00:01:37.503>of <00:01:37.762>there <00:01:38.021>were <00:01:38.280>many.

36
00:01:39.655 --> 00:01:43.755
<00:01:39.655>So, <00:01:39.782>first <00:01:39.909>of <00:01:40.036>all, <00:01:40.164>what <00:01:40.291>is <00:01:40.418>Kafka <00:01:40.545>and <00:01:40.673>what <00:01:40.800>are <00:01:40.927>Kafka <00:01:41.055>streams?

37
00:01:44.925 --> 00:01:47.715
<00:01:44.925>Kafka <00:01:45.158>is <00:01:45.391>a <00:01:45.625>distributed <00:01:45.858>event <00:01:46.091>streaming <00:01:46.325>platform.

38
00:01:48.175 --> 00:01:52.435
<00:01:48.175>So <00:01:48.315>it <00:01:48.455>basically <00:01:48.595>consists <00:01:48.735>of <00:01:48.875>topics <00:01:49.015>where <00:01:49.155>you <00:01:49.295>can <00:01:49.435>store <00:01:49.575>data

39
00:01:52.855 --> 00:01:55.275
<00:01:52.854>as <00:01:53.054>key <00:01:53.255>value <00:01:53.454>pairs <00:01:53.654>in <00:01:53.854>a <00:01:54.054>commit <00:01:54.255>log.

40
00:01:55.655 --> 00:01:58.875
<00:01:55.655>So <00:01:55.782>here <00:01:55.909>on <00:01:56.036>the <00:01:56.164>right <00:01:56.291>you <00:01:56.418>see <00:01:56.545>this, <00:01:56.673>uh, <00:01:56.800>these <00:01:56.927>numbered <00:01:57.055>items.

41
00:01:59.285 --> 00:02:00.875
<00:01:59.285>These <00:01:59.565>are <00:01:59.845>records <00:02:00.125>in <00:02:00.405>a <00:02:00.685>topic.

42
00:02:01.665 --> 00:02:04.075
<00:02:01.665>Then <00:02:01.898>you <00:02:02.131>can <00:02:02.365>have <00:02:02.598>producers <00:02:02.831>writing <00:02:03.065>data

43
00:02:04.295 --> 00:02:05.715
<00:02:04.295>to <00:02:04.550>the <00:02:04.806>end <00:02:05.061>of <00:02:05.317>the <00:02:05.573>topic.

44
00:02:06.015 --> 00:02:07.995
<00:02:06.015>So <00:02:06.248>this <00:02:06.481>is <00:02:06.715>immutable <00:02:06.948>data <00:02:07.181>being <00:02:07.415>appended,

45
00:02:08.855 --> 00:02:10.155
<00:02:08.854>and <00:02:09.089>then <00:02:09.323>you <00:02:09.556>can <00:02:09.791>have <00:02:10.025>one

46
00:02:10.155 --> 00:02:13.075
<00:02:10.155>or <00:02:10.330>more <00:02:10.505>consumers <00:02:10.680>reading <00:02:10.854>the <00:02:11.030>data <00:02:11.205>in <00:02:11.380>the <00:02:11.555>topic

47
00:02:13.755 --> 00:02:14.795
<00:02:13.755>independently <00:02:14.067>from <00:02:14.379>each <00:02:14.690>other.

48
00:02:15.175 --> 00:02:17.995
<00:02:15.175>And <00:02:15.375>each <00:02:15.575>consumer <00:02:15.775>has <00:02:15.975>its <00:02:16.175>own <00:02:16.375>consumer <00:02:16.575>offset,

49
00:02:18.005 --> 00:02:20.075
<00:02:18.005>which <00:02:18.204>is <00:02:18.405>basically <00:02:18.605>a <00:02:18.805>cursor <00:02:19.005>into <00:02:19.204>the <00:02:19.405>topic,

50
00:02:20.225 --> 00:02:21.915
<00:02:20.225>telling <00:02:20.505>it <00:02:20.785>where <00:02:21.065>to <00:02:21.345>read <00:02:21.625>from.

51
00:02:22.105 --> 00:02:24.995
<00:02:22.105>Usually <00:02:22.260>they <00:02:22.416>read <00:02:22.571>from, <00:02:22.727>uh, <00:02:22.882>the <00:02:23.038>start <00:02:23.193>to <00:02:23.349>the <00:02:23.505>end,

52
00:02:25.255 --> 00:02:27.595
<00:02:25.255>and <00:02:25.430>once <00:02:25.605>they <00:02:25.780>reach <00:02:25.954>the <00:02:26.130>end, <00:02:26.305>uh, <00:02:26.480>they <00:02:26.655>wait

53
00:02:27.595 --> 00:02:29.515
<00:02:27.595>until <00:02:27.828>the <00:02:28.061>producer <00:02:28.295>writes <00:02:28.528>a <00:02:28.761>new <00:02:28.995>record.

54
00:02:29.615 --> 00:02:31.675
<00:02:29.615>So <00:02:29.895>it's <00:02:30.175>essentially <00:02:30.455>published <00:02:30.735>and <00:02:31.015>subscribe.

55
00:02:32.455 --> 00:02:35.275
<00:02:32.454>Uh, <00:02:32.610>the <00:02:32.766>Kafka <00:02:32.921>topics <00:02:33.077>in <00:02:33.232>this <00:02:33.388>talk <00:02:33.543>will <00:02:33.699>be <00:02:33.855>represented

56
00:02:35.415 --> 00:02:36.995
<00:02:35.415>by <00:02:35.881>these <00:02:36.348>green <00:02:36.815>cylinders.

57
00:02:38.485 --> 00:02:42.435
<00:02:38.485>Physically, <00:02:38.718>the <00:02:38.951>partitions <00:02:39.185>are <00:02:39.418>split, <00:02:39.651>oh, <00:02:39.885>sorry.

58
00:02:42.435 --> 00:02:44.595
<00:02:42.435>The <00:02:42.668>topics <00:02:42.901>are <00:02:43.135>split <00:02:43.368>into <00:02:43.601>multiple <00:02:43.835>partitions.

59
00:02:44.975 --> 00:02:47.355
<00:02:44.975>And <00:02:45.130>in <00:02:45.286>this <00:02:45.441>case, <00:02:45.597>if <00:02:45.752>we <00:02:45.908>now <00:02:46.063>consider <00:02:46.219>these <00:02:46.375>numbers

60
00:02:47.575 --> 00:02:52.395
<00:02:47.575>to <00:02:47.691>be <00:02:47.808>the <00:02:47.924>keys <00:02:48.041>of <00:02:48.158>the <00:02:48.274>key <00:02:48.391>value <00:02:48.508>payers, <00:02:48.625>uh, <00:02:48.741>in <00:02:48.858>the <00:02:48.975>split,

61
00:02:52.465 --> 00:02:56.035
<00:02:52.465>they're <00:02:52.640>being <00:02:52.815>put <00:02:52.990>into <00:02:53.165>partition <00:02:53.340>one <00:02:53.515>when <00:02:53.690>they're, <00:02:53.865>uh,

62
00:02:56.175 --> 00:02:57.715
<00:02:56.175>odd <00:02:56.637>valued <00:02:57.099>or <00:02:57.561>partition.

63
00:02:57.715 --> 00:02:58.915
<00:02:57.715>Two, <00:02:57.985>when <00:02:58.255>they're <00:02:58.525>even <00:02:58.795>valued.

64
00:03:00.115 --> 00:03:02.885
<00:03:00.115>Then <00:03:00.290>you <00:03:00.465>can <00:03:00.640>have <00:03:00.815>a <00:03:00.990>consumer <00:03:01.165>group <00:03:01.340>that, <00:03:01.515>uh,

65
00:03:03.135 --> 00:03:04.285
<00:03:03.135>reads <00:03:03.652>the <00:03:04.170>partitions.

66
00:03:04.905 --> 00:03:08.245
<00:03:04.905>Uh, <00:03:05.045>you <00:03:05.185>can <00:03:05.325>have <00:03:05.465>one <00:03:05.605>or <00:03:05.745>more <00:03:05.885>consumers <00:03:06.025>in <00:03:06.165>that <00:03:06.305>group,

67
00:03:08.305 --> 00:03:10.965
<00:03:08.305>and <00:03:08.445>each <00:03:08.585>one <00:03:08.725>can <00:03:08.865>be <00:03:09.005>assigned <00:03:09.145>to <00:03:09.285>one <00:03:09.425>or <00:03:09.565>more <00:03:09.705>partitions.

68
00:03:11.665 --> 00:03:14.045
<00:03:11.665>So <00:03:11.945>they <00:03:12.225>read <00:03:12.505>those <00:03:12.785>partitions <00:03:13.065>independently,

69
00:03:14.185 --> 00:03:15.285
<00:03:14.185>but <00:03:14.350>as <00:03:14.515>a <00:03:14.680>whole, <00:03:14.845>as <00:03:15.010>a <00:03:15.175>group,

70
00:03:15.475 --> 00:03:17.965
<00:03:15.475>they <00:03:15.674>read <00:03:15.875>all <00:03:16.075>the <00:03:16.275>data <00:03:16.475>in <00:03:16.674>the <00:03:16.875>topic.

71
00:03:18.545 --> 00:03:20.965
<00:03:18.545>And <00:03:18.825>this <00:03:19.105>allows <00:03:19.385>for <00:03:19.665>horizontal <00:03:19.945>scalability

72
00:03:21.145 --> 00:03:23.805
<00:03:21.145>by <00:03:21.378>increasing <00:03:21.611>the <00:03:21.845>parallelism <00:03:22.078>of <00:03:22.311>the <00:03:22.545>system.

73
00:03:25.545 --> 00:03:29.685
<00:03:25.545>So <00:03:25.685>Kafka <00:03:25.825>streams <00:03:25.965>now <00:03:26.105>are <00:03:26.245>ba <00:03:26.385>basically <00:03:26.525>Kafka <00:03:26.665>streams <00:03:26.805>is <00:03:26.945>a

74
00:03:29.685 --> 00:03:31.085
<00:03:29.685>domain <00:03:30.315>specific <00:03:30.945>language,

75
00:03:31.585 --> 00:03:34.285
<00:03:31.585>and <00:03:31.818>it's <00:03:32.051>an <00:03:32.285>abstraction <00:03:32.518>over <00:03:32.751>the <00:03:32.985>consumers

76
00:03:34.345 --> 00:03:37.485
<00:03:34.345>and <00:03:34.578>producers <00:03:34.811>in <00:03:35.045>Kafka <00:03:35.278>that <00:03:35.511>allow <00:03:35.745>you

77
00:03:37.485 --> 00:03:39.045
<00:03:37.485>to <00:03:38.185>describe <00:03:38.885>topologies.

78
00:03:39.905 --> 00:03:43.885
<00:03:39.905>So <00:03:40.032>a <00:03:40.159>topology <00:03:40.286>is <00:03:40.414>input <00:03:40.541>topics <00:03:40.668>On <00:03:40.795>the <00:03:40.923>right, <00:03:41.050>you <00:03:41.177>see <00:03:41.305>here,

79
00:03:43.885 --> 00:03:45.805
<00:03:43.885>there <00:03:44.084>are <00:03:44.285>two <00:03:44.484>input <00:03:44.685>topics <00:03:44.885>in <00:03:45.084>this <00:03:45.285>topology.

80
00:03:46.385 --> 00:03:49.485
<00:03:46.385>Uh, <00:03:46.540>you <00:03:46.696>have <00:03:46.851>operators <00:03:47.007>such <00:03:47.162>as <00:03:47.318>map, <00:03:47.473>join <00:03:47.629>and <00:03:47.785>filter,

81
00:03:49.745 --> 00:03:51.085
<00:03:49.745>and <00:03:49.986>then <00:03:50.227>you <00:03:50.468>have <00:03:50.709>output <00:03:50.951>topics.

82
00:03:51.985 --> 00:03:55.445
<00:03:51.985>And <00:03:52.125>you <00:03:52.265>can <00:03:52.405>view <00:03:52.545>a <00:03:52.685>topic <00:03:52.825>as <00:03:52.965>either <00:03:53.105>a <00:03:53.245>K <00:03:53.385>stream,

83
00:03:55.895 --> 00:03:58.685
<00:03:55.895>which <00:03:56.050>is <00:03:56.206>basically <00:03:56.361>just <00:03:56.517>a <00:03:56.672>stream <00:03:56.828>of <00:03:56.983>data <00:03:57.139>points, <00:03:57.295>one

84
00:03:58.685 --> 00:04:00.845
<00:03:58.685>after <00:03:58.885>the <00:03:59.085>other <00:03:59.285>that <00:03:59.485>go <00:03:59.685>through <00:03:59.885>the <00:04:00.085>topology.

85
00:04:01.425 --> 00:04:04.845
<00:04:01.425>Or <00:04:01.565>you <00:04:01.705>can <00:04:01.845>view <00:04:01.985>a <00:04:02.125>topic <00:04:02.265>as <00:04:02.405>a <00:04:02.545>K <00:04:02.685>table, <00:04:02.825>um,

86
00:04:04.975 --> 00:04:06.245
<00:04:04.975>which <00:04:05.356>basically <00:04:05.737>allows <00:04:06.118>you

87
00:04:06.245 --> 00:04:09.205
<00:04:06.245>to <00:04:06.400>do <00:04:06.556>lookups <00:04:06.711>in <00:04:06.867>the <00:04:07.022>topic <00:04:07.178>based <00:04:07.333>on <00:04:07.489>a <00:04:07.645>key.

88
00:04:09.465 --> 00:04:12.485
<00:04:09.465>And <00:04:09.620>you <00:04:09.776>can <00:04:09.931>then <00:04:10.087>join <00:04:10.242>these, <00:04:10.398>for <00:04:10.553>example, <00:04:10.709>to <00:04:10.865>streams

89
00:04:12.485 --> 00:04:14.725
<00:04:12.485>to <00:04:12.718>enrich <00:04:12.951>data <00:04:13.185>in <00:04:13.418>your <00:04:13.651>data <00:04:13.885>streams.

90
00:04:15.425 --> 00:04:16.565
<00:04:15.425>All <00:04:15.630>right, <00:04:15.835>I <00:04:16.040>think <00:04:16.245>that's, <00:04:16.451>uh,

91
00:04:16.715 --> 00:04:19.365
<00:04:16.714>hopefully <00:04:16.994>enough <00:04:17.274>introduction <00:04:17.554>and <00:04:17.834>uh, <00:04:18.114>Dennis.

92
00:04:19.915 --> 00:04:21.605
<00:04:19.915>Alright, <00:04:20.195>so <00:04:20.475>how <00:04:20.755>do <00:04:21.035>we <00:04:21.315>get

93
00:04:21.605 --> 00:04:23.765
<00:04:21.605>to <00:04:21.885>actually <00:04:22.165>build <00:04:22.445>Kafka <00:04:22.725>streams <00:04:23.005>microservices?

94
00:04:24.635 --> 00:04:27.285
<00:04:24.635>Well, <00:04:24.762>it <00:04:24.889>all <00:04:25.016>started <00:04:25.144>with <00:04:25.271>a <00:04:25.398>brief <00:04:25.525>from <00:04:25.653>our <00:04:25.780>client <00:04:25.907>who <00:04:26.034>wanted

95
00:04:27.285 --> 00:04:30.445
<00:04:27.285>to <00:04:27.485>build <00:04:27.685>a <00:04:27.885>customer <00:04:28.085>applic <00:04:28.285>customer <00:04:28.485>facing <00:04:28.685>application

96
00:04:30.445 --> 00:04:33.805
<00:04:30.445>with <00:04:30.600>us, <00:04:30.756>but <00:04:30.911>didn't <00:04:31.067>want <00:04:31.222>to <00:04:31.378>build <00:04:31.533>yet <00:04:31.689>another <00:04:31.844>monolith

97
00:04:33.805 --> 00:04:35.685
<00:04:33.805>and <00:04:33.960>data <00:04:34.116>silo <00:04:34.271>as <00:04:34.427>they <00:04:34.582>have <00:04:34.738>done <00:04:34.893>in <00:04:35.049>the <00:04:35.205>past.

98
00:04:35.945 --> 00:04:40.085
<00:04:35.945>So <00:04:36.085>they <00:04:36.224>asked <00:04:36.365>us <00:04:36.505>to <00:04:36.645>build <00:04:36.784>actually <00:04:36.925>a <00:04:37.065>set <00:04:37.205>of <00:04:37.344>microservices

99
00:04:40.235 --> 00:04:43.765
<00:04:40.235>that <00:04:40.362>not <00:04:40.489>only <00:04:40.616>we, <00:04:40.744>but <00:04:40.871>also <00:04:40.998>other <00:04:41.125>teams <00:04:41.253>in <00:04:41.380>the <00:04:41.507>company <00:04:41.635>could

100
00:04:43.825 --> 00:04:47.365
<00:04:43.825>use, <00:04:44.025>uh, <00:04:44.224>to <00:04:44.425>create <00:04:44.625>an <00:04:44.825>ecosystem <00:04:45.025>application <00:04:45.224>that,

101
00:04:47.385 --> 00:04:48.605
<00:04:47.385>uh, <00:04:47.934>tightly <00:04:48.483>integrated.

102
00:04:49.825 --> 00:04:52.165
<00:04:49.825>And <00:04:50.058>for <00:04:50.291>that <00:04:50.525>we <00:04:50.758>had <00:04:50.991>multiple <00:04:51.224>constraints.

103
00:04:52.185 --> 00:04:56.605
<00:04:52.185>We <00:04:52.301>had <00:04:52.418>to <00:04:52.535>work <00:04:52.651>with <00:04:52.768>one <00:04:52.885>of <00:04:53.001>them <00:04:53.118>being <00:04:53.235>that <00:04:53.351>we <00:04:53.468>had <00:04:53.585>multiple

104
00:04:57.365 --> 00:04:59.045
<00:04:57.365>services, <00:04:57.645>existing <00:04:57.925>services <00:04:58.205>in <00:04:58.485>the <00:04:58.765>company

105
00:04:59.115 --> 00:05:00.685
<00:04:59.115>that <00:04:59.395>we <00:04:59.675>had <00:04:59.955>to <00:05:00.235>integrate <00:05:00.515>with.

106
00:05:01.385 --> 00:05:03.685
<00:05:01.385>And <00:05:01.664>several <00:05:01.945>of <00:05:02.224>these <00:05:02.505>were <00:05:02.784>deployed

107
00:05:03.785 --> 00:05:07.325
<00:05:03.785>to <00:05:03.960>on-premise <00:05:04.135>data <00:05:04.310>centers, <00:05:04.485>meaning <00:05:04.660>that <00:05:04.835>they <00:05:05.010>were <00:05:05.185>not

108
00:05:07.465 --> 00:05:10.205
<00:05:07.465>as <00:05:07.664>available <00:05:07.864>as <00:05:08.065>the <00:05:08.265>services <00:05:08.465>that <00:05:08.664>we <00:05:08.864>were

109
00:05:10.205 --> 00:05:11.885
<00:05:10.205>to <00:05:10.484>deploy <00:05:10.765>to <00:05:11.044>the <00:05:11.325>public <00:05:11.604>cloud.

110
00:05:12.025 --> 00:05:15.165
<00:05:12.025>So <00:05:12.180>we <00:05:12.336>had <00:05:12.491>to <00:05:12.647>think <00:05:12.802>about <00:05:12.958>how <00:05:13.113>do <00:05:13.269>we <00:05:13.424>integrate

111
00:05:15.165 --> 00:05:17.325
<00:05:15.165>with <00:05:15.445>these <00:05:15.725>services <00:05:16.005>without <00:05:16.285>compromising <00:05:16.565>the

112
00:05:17.325 --> 00:05:18.845
<00:05:17.325>availability <00:05:17.781>of <00:05:18.237>our <00:05:18.693>system.

113
00:05:20.265 --> 00:05:25.085
<00:05:20.265>And <00:05:20.405>also <00:05:20.544>the <00:05:20.685>client <00:05:20.825>wanted <00:05:20.965>to <00:05:21.104>establish <00:05:21.245>Kafka <00:05:21.385>as <00:05:21.525>this <00:05:21.664>system

114
00:05:25.085 --> 00:05:27.805
<00:05:25.085>of <00:05:25.364>choice <00:05:25.645>to <00:05:25.924>integrate <00:05:26.205>the <00:05:26.484>different

115
00:05:27.955 --> 00:05:29.245
<00:05:27.955>systems <00:05:28.342>in <00:05:28.729>their <00:05:29.116>company.

116
00:05:29.345 --> 00:05:30.845
<00:05:29.345>So <00:05:29.513>we <00:05:29.682>had <00:05:29.851>to <00:05:30.020>connect <00:05:30.188>to <00:05:30.357>that <00:05:30.526>as <00:05:30.695>well.

117
00:05:31.785 --> 00:05:34.965
<00:05:31.785>So <00:05:31.960>we <00:05:32.135>derived <00:05:32.310>three <00:05:32.485>central <00:05:32.660>goals <00:05:32.835>for <00:05:33.010>our <00:05:33.185>architecture.

118
00:05:35.215 --> 00:05:37.965
<00:05:35.215>First <00:05:35.390>one <00:05:35.565>being <00:05:35.740>Kafka <00:05:35.915>integration, <00:05:36.090>as <00:05:36.265>I <00:05:36.440>just <00:05:36.615>mentioned.

119
00:05:38.665 --> 00:05:41.165
<00:05:38.665>Uh, <00:05:38.898>the <00:05:39.131>second <00:05:39.365>robustness <00:05:39.598>and <00:05:39.831>failure <00:05:40.065>tolerance.

120
00:05:41.265 --> 00:05:44.005
<00:05:41.265>We <00:05:41.420>knew <00:05:41.576>that <00:05:41.731>multiple <00:05:41.887>applications <00:05:42.042>will <00:05:42.198>be <00:05:42.353>built <00:05:42.509>on <00:05:42.664>this,

121
00:05:44.555 --> 00:05:46.045
<00:05:44.555>that <00:05:44.890>would <00:05:45.225>be <00:05:45.560>customer <00:05:45.896>facing.

122
00:05:46.145 --> 00:05:49.325
<00:05:46.145>So <00:05:46.284>we <00:05:46.424>needed <00:05:46.565>the <00:05:46.705>platform <00:05:46.844>to <00:05:46.984>be <00:05:47.125>as <00:05:47.265>robust <00:05:47.405>as <00:05:47.544>possible,

123
00:05:50.225 --> 00:05:53.245
<00:05:50.225>and <00:05:50.380>we <00:05:50.536>also <00:05:50.691>wanted <00:05:50.847>to <00:05:51.002>make <00:05:51.158>the <00:05:51.313>whole <00:05:51.469>system <00:05:51.625>extensible.

124
00:05:53.305 --> 00:05:56.205
<00:05:53.305>So <00:05:53.480>any <00:05:53.655>team <00:05:53.830>in <00:05:54.005>the <00:05:54.180>company <00:05:54.355>should <00:05:54.530>be <00:05:54.705>able

125
00:05:56.205 --> 00:05:57.565
<00:05:56.205>to <00:05:56.510>define <00:05:56.817>a <00:05:57.123>new <00:05:57.429>service

126
00:05:58.105 --> 00:06:02.325
<00:05:58.105>and <00:05:58.305>integrate <00:05:58.505>into <00:05:58.705>the <00:05:58.905>existing <00:05:59.105>microservice <00:05:59.305>ecosystem <00:05:59.505>with

127
00:06:02.545 --> 00:06:05.645
<00:06:02.545>as <00:06:02.745>little <00:06:02.945>central <00:06:03.145>coordination <00:06:03.345>needed <00:06:03.545>as <00:06:03.745>possible <00:06:03.945>so

128
00:06:05.645 --> 00:06:08.325
<00:06:05.645>that <00:06:05.878>the <00:06:06.111>whole <00:06:06.344>platform <00:06:06.578>can <00:06:06.811>grow <00:06:07.044>organically.

129
00:06:09.465 --> 00:06:13.045
<00:06:09.465>So <00:06:09.592>from <00:06:09.719>these <00:06:09.846>goals, <00:06:09.974>we <00:06:10.101>set <00:06:10.228>out <00:06:10.355>to <00:06:10.483>create <00:06:10.610>a <00:06:10.737>first <00:06:10.864>design,

130
00:06:13.815 --> 00:06:15.845
<00:06:13.815>which <00:06:14.048>look <00:06:14.281>pretty <00:06:14.515>much <00:06:14.748>like <00:06:14.981>a <00:06:15.215>classic

131
00:06:16.165 --> 00:06:17.845
<00:06:16.165>microservice <00:06:17.565>architecture.

132
00:06:17.905 --> 00:06:21.685
<00:06:17.905>So <00:06:18.104>we <00:06:18.304>define <00:06:18.505>microservices <00:06:18.705>around <00:06:18.905>the <00:06:19.104>business <00:06:19.304>domains,

133
00:06:22.305 --> 00:06:25.365
<00:06:22.305>and <00:06:22.480>those <00:06:22.655>microservices <00:06:22.830>each <00:06:23.005>have <00:06:23.180>their <00:06:23.355>own <00:06:23.530>private <00:06:23.705>database

134
00:06:25.625 --> 00:06:28.805
<00:06:25.625>and <00:06:25.780>send <00:06:25.936>commands <00:06:26.091>and <00:06:26.247>queries <00:06:26.402>to <00:06:26.558>each <00:06:26.713>other <00:06:26.869>as <00:06:27.025>needed.

135
00:06:29.465 --> 00:06:33.805
<00:06:29.465>The <00:06:29.592>role <00:06:29.719>of <00:06:29.846>Kafka <00:06:29.974>here <00:06:30.101>would <00:06:30.228>have <00:06:30.355>been <00:06:30.483>only <00:06:30.610>that <00:06:30.737>of, <00:06:30.864>uh,

136
00:06:34.195 --> 00:06:37.445
<00:06:34.195>basically <00:06:34.370>an <00:06:34.545>event <00:06:34.719>and <00:06:34.895>message <00:06:35.070>queue <00:06:35.245>between <00:06:35.420>our <00:06:35.594>system

137
00:06:37.545 --> 00:06:39.405
<00:06:37.545>and <00:06:37.778>the <00:06:38.011>other <00:06:38.245>systems <00:06:38.478>in <00:06:38.711>the <00:06:38.945>company,

138
00:06:40.055 --> 00:06:41.845
<00:06:40.055>which <00:06:40.405>also <00:06:40.755>had <00:06:41.105>the <00:06:41.455>benefit

139
00:06:42.185 --> 00:06:46.485
<00:06:42.185>of <00:06:42.360>decoupling <00:06:42.535>ourselves <00:06:42.710>a <00:06:42.885>bit <00:06:43.060>from <00:06:43.235>their <00:06:43.410>availability, <00:06:43.585>um,

140
00:06:46.485 --> 00:06:48.925
<00:06:46.485>because <00:06:46.718>the <00:06:46.951>communication <00:06:47.185>to <00:06:47.418>Kafka <00:06:47.651>is <00:06:47.885>synchronous.

141
00:06:49.945 --> 00:06:52.005
<00:06:49.945>So <00:06:50.224>this <00:06:50.505>design <00:06:50.784>is <00:06:51.065>pretty <00:06:51.344>straightforward,

142
00:06:52.265 --> 00:06:55.085
<00:06:52.265>but <00:06:52.420>we <00:06:52.576>were <00:06:52.731>a <00:06:52.887>bit <00:06:53.042>concerned <00:06:53.198>about <00:06:53.353>the <00:06:53.509>direct <00:06:53.664>calls

143
00:06:55.085 --> 00:06:56.365
<00:06:55.085>between <00:06:55.661>our <00:06:56.237>services

144
00:06:56.395 --> 00:07:00.205
<00:06:56.395>because <00:06:56.594>it <00:06:56.794>means <00:06:56.995>that <00:06:57.195>there's <00:06:57.395>quite <00:06:57.594>tight <00:06:57.794>coupling

145
00:07:00.205 --> 00:07:01.685
<00:07:00.205>between <00:07:00.471>these <00:07:00.737>services <00:07:01.004>and <00:07:01.270>it's <00:07:01.537>hard

146
00:07:01.705 --> 00:07:03.965
<00:07:01.705>to <00:07:01.938>create <00:07:02.171>a <00:07:02.405>new <00:07:02.638>service <00:07:02.871>which <00:07:03.104>integrates

147
00:07:03.965 --> 00:07:05.565
<00:07:03.965>with <00:07:04.244>the <00:07:04.525>existing <00:07:04.804>ones, <00:07:05.085>which <00:07:05.364>is

148
00:07:05.565 --> 00:07:06.765
<00:07:05.565>what <00:07:05.745>we <00:07:05.925>actually <00:07:06.105>want <00:07:06.284>to <00:07:06.465>make <00:07:06.645>easy.

149
00:07:07.625 --> 00:07:09.925
<00:07:07.625>We <00:07:07.825>were <00:07:08.025>also <00:07:08.225>a <00:07:08.425>bit <00:07:08.625>concerned <00:07:08.825>about <00:07:09.025>the

150
00:07:10.075 --> 00:07:11.405
<00:07:10.075>consistency <00:07:10.474>of <00:07:10.873>the <00:07:11.272>system.

151
00:07:11.625 --> 00:07:15.485
<00:07:11.625>If <00:07:11.780>some <00:07:11.936>user <00:07:12.091>action <00:07:12.247>actually <00:07:12.402>requires <00:07:12.558>us <00:07:12.713>to <00:07:12.869>send <00:07:13.025>commands

152
00:07:15.485 --> 00:07:16.645
<00:07:15.485>to <00:07:16.007>multiple <00:07:16.529>microservices,

153
00:07:17.155 --> 00:07:18.645
<00:07:17.155>what <00:07:17.346>happens <00:07:17.538>when <00:07:17.729>one <00:07:17.921>of <00:07:18.112>them <00:07:18.304>is <00:07:18.496>down?

154
00:07:19.785 --> 00:07:23.685
<00:07:19.785>So <00:07:19.940>then <00:07:20.096>we <00:07:20.251>thought, <00:07:20.407>what <00:07:20.562>about <00:07:20.718>actually <00:07:20.873>using <00:07:21.029>Kafka <00:07:21.185>also

155
00:07:23.705 --> 00:07:25.565
<00:07:23.705>for <00:07:23.984>the <00:07:24.265>communication <00:07:24.544>within <00:07:24.825>our <00:07:25.104>system?

156
00:07:26.265 --> 00:07:29.405
<00:07:26.265>And <00:07:26.405>we <00:07:26.544>came <00:07:26.685>up <00:07:26.825>with <00:07:26.965>this <00:07:27.104>more <00:07:27.245>event <00:07:27.385>driven <00:07:27.525>approach <00:07:27.664>where

157
00:07:30.065 --> 00:07:32.285
<00:07:30.065>the <00:07:30.265>services <00:07:30.465>wouldn't <00:07:30.665>send <00:07:30.865>commands <00:07:31.065>to <00:07:31.265>each <00:07:31.465>other,

158
00:07:32.385 --> 00:07:34.485
<00:07:32.385>but <00:07:32.664>instead <00:07:32.945>they <00:07:33.224>would <00:07:33.505>publish <00:07:33.784>events

159
00:07:34.635 --> 00:07:36.445
<00:07:34.635>that <00:07:34.914>then <00:07:35.195>other <00:07:35.474>services <00:07:35.755>can <00:07:36.034>read.

160
00:07:37.025 --> 00:07:41.085
<00:07:37.025>So <00:07:37.180>now <00:07:37.336>our <00:07:37.491>services <00:07:37.647>are <00:07:37.802>less <00:07:37.958>coupled <00:07:38.113>to <00:07:38.269>each <00:07:38.424>other

161
00:07:41.225 --> 00:07:44.205
<00:07:41.225>and <00:07:41.365>it's <00:07:41.505>easy <00:07:41.645>to <00:07:41.785>add <00:07:41.925>a <00:07:42.065>new <00:07:42.205>service <00:07:42.345>that <00:07:42.485>listens <00:07:42.625>to

162
00:07:44.225 --> 00:07:45.645
<00:07:44.225>and <00:07:44.544>reacts <00:07:44.864>to <00:07:45.183>existing <00:07:45.503>events,

163
00:07:45.775 --> 00:07:47.445
<00:07:45.775>which <00:07:46.054>is <00:07:46.335>a <00:07:46.614>plus <00:07:46.895>for <00:07:47.174>extensibility.

164
00:07:48.345 --> 00:07:52.765
<00:07:48.345>It <00:07:48.472>also <00:07:48.599>means <00:07:48.726>that <00:07:48.854>if <00:07:48.981>one <00:07:49.108>of <00:07:49.235>our <00:07:49.363>services <00:07:49.490>is <00:07:49.617>down, <00:07:49.745>uh,

165
00:07:52.835 --> 00:07:54.245
<00:07:52.835>once <00:07:53.088>it <00:07:53.342>comes <00:07:53.596>back <00:07:53.850>up <00:07:54.104>again,

166
00:07:54.245 --> 00:07:55.805
<00:07:54.245>it <00:07:54.595>can <00:07:54.945>just <00:07:55.295>continue <00:07:55.645>processing

167
00:07:55.805 --> 00:07:56.965
<00:07:55.805>events <00:07:56.013>from <00:07:56.222>where <00:07:56.431>it <00:07:56.640>left <00:07:56.849>off.

168
00:07:58.225 --> 00:08:02.845
<00:07:58.225>But <00:07:58.365>we <00:07:58.505>still <00:07:58.645>had <00:07:58.785>the <00:07:58.925>problem <00:07:59.065>that <00:07:59.205>if <00:07:59.345>a <00:07:59.485>service <00:07:59.625>has

169
00:08:02.845 --> 00:08:03.845
<00:08:02.845>to <00:08:03.145>process <00:08:03.445>an <00:08:03.745>event

170
00:08:04.025 --> 00:08:05.845
<00:08:04.025>and <00:08:04.200>needs <00:08:04.375>some <00:08:04.549>additional <00:08:04.724>data <00:08:04.900>from <00:08:05.075>some <00:08:05.250>other <00:08:05.424>service,

171
00:08:06.425 --> 00:08:09.285
<00:08:06.425>it <00:08:06.565>still <00:08:06.705>needs <00:08:06.845>to <00:08:06.985>do <00:08:07.125>a <00:08:07.265>direct <00:08:07.405>query <00:08:07.545>to <00:08:07.685>that <00:08:07.825>service.

172
00:08:09.425 --> 00:08:12.125
<00:08:09.425>So <00:08:09.658>there <00:08:09.891>is <00:08:10.125>still <00:08:10.358>potentially <00:08:10.591>availability <00:08:10.825>problem.

173
00:08:13.705 --> 00:08:15.645
<00:08:13.705>So <00:08:13.905>we <00:08:14.104>thought <00:08:14.305>a <00:08:14.505>bit <00:08:14.705>more <00:08:14.905>about <00:08:15.104>this.

174
00:08:15.665 --> 00:08:19.125
<00:08:15.665>We <00:08:15.820>did <00:08:15.976>some <00:08:16.131>research <00:08:16.287>and <00:08:16.442>we <00:08:16.598>came <00:08:16.753>across <00:08:16.909>this <00:08:17.065>book

175
00:08:19.675 --> 00:08:22.525
<00:08:19.675>designing <00:08:19.908>event <00:08:20.141>room <00:08:20.375>systems <00:08:20.608>from <00:08:20.841>Ben <00:08:21.075>Stockport.

176
00:08:23.345 --> 00:08:26.165
<00:08:23.345>And <00:08:23.545>that <00:08:23.745>book <00:08:23.945>had <00:08:24.145>two <00:08:24.345>central <00:08:24.545>ideas <00:08:24.745>which

177
00:08:26.165 --> 00:08:27.325
<00:08:26.165>deeply <00:08:26.687>influence <00:08:27.209>us.

178
00:08:28.065 --> 00:08:29.405
<00:08:28.065>The <00:08:28.366>first <00:08:28.668>is <00:08:28.969>the <00:08:29.270>idea

179
00:08:29.475 --> 00:08:32.845
<00:08:29.475>that <00:08:29.708>we <00:08:29.941>could <00:08:30.175>actually <00:08:30.408>treat <00:08:30.641>event <00:08:30.875>streams

180
00:08:33.185 --> 00:08:34.525
<00:08:33.184>as <00:08:33.587>a <00:08:33.988>primary <00:08:34.390>form

181
00:08:34.525 --> 00:08:38.565
<00:08:34.525>of <00:08:34.641>storage <00:08:34.758>if <00:08:34.875>we <00:08:34.991>store <00:08:35.108>them <00:08:35.225>in <00:08:35.341>Kafka <00:08:35.458>topics <00:08:35.574>that <00:08:35.691>we <00:08:35.808>set <00:08:35.924>up

182
00:08:38.565 --> 00:08:39.565
<00:08:38.565>to <00:08:39.015>be <00:08:39.465>persistent.

183
00:08:39.585 --> 00:08:42.805
<00:08:39.585>So <00:08:39.865>then <00:08:40.145>these <00:08:40.425>events <00:08:40.705>basically <00:08:40.985>describe

184
00:08:43.385 --> 00:08:44.685
<00:08:43.385>the <00:08:43.619>whole <00:08:43.852>state <00:08:44.087>of <00:08:44.320>the <00:08:44.555>system.

185
00:08:45.305 --> 00:08:47.005
<00:08:45.305>You <00:08:45.505>might <00:08:45.704>know <00:08:45.905>this <00:08:46.104>idea <00:08:46.305>as <00:08:46.505>event <00:08:46.704>sourcing.

186
00:08:48.145 --> 00:08:49.645
<00:08:48.145>And <00:08:48.482>the <00:08:48.819>second <00:08:49.157>idea <00:08:49.495>is

187
00:08:49.755 --> 00:08:51.925
<00:08:49.755>that <00:08:49.988>once <00:08:50.221>you <00:08:50.455>have <00:08:50.688>these <00:08:50.921>event <00:08:51.155>streams,

188
00:08:51.985 --> 00:08:56.005
<00:08:51.985>you <00:08:52.125>can <00:08:52.265>actually <00:08:52.405>reduce <00:08:52.545>them <00:08:52.685>in <00:08:52.825>real <00:08:52.965>time <00:08:53.105>to <00:08:53.245>any <00:08:53.385>kind

189
00:08:56.005 --> 00:08:58.005
<00:08:56.005>of <00:08:56.238>query <00:08:56.471>or <00:08:56.705>perform <00:08:56.938>that <00:08:57.171>you <00:08:57.405>need

190
00:08:58.005 --> 00:08:59.205
<00:08:58.005>for <00:08:58.275>your <00:08:58.545>specific <00:08:58.815>use <00:08:59.085>case.

191
00:08:59.865 --> 00:09:02.645
<00:08:59.865>So <00:09:00.020>you <00:09:00.176>might <00:09:00.331>stream <00:09:00.487>the <00:09:00.642>data <00:09:00.798>to <00:09:00.953>a <00:09:01.109>relational <00:09:01.265>database

192
00:09:02.645 --> 00:09:04.245
<00:09:02.645>to <00:09:02.878>do <00:09:03.111>relational <00:09:03.345>queries <00:09:03.578>on <00:09:03.811>the <00:09:04.045>data,

193
00:09:05.025 --> 00:09:07.645
<00:09:05.025>or <00:09:05.199>you <00:09:05.375>might <00:09:05.550>stream <00:09:05.725>them <00:09:05.900>to <00:09:06.074>something <00:09:06.250>like <00:09:06.425>Elasticsearch

194
00:09:07.705 --> 00:09:08.725
<00:09:07.705>to <00:09:07.934>have <00:09:08.164>full <00:09:08.393>text <00:09:08.623>search.

195
00:09:10.345 --> 00:09:13.205
<00:09:10.345>And <00:09:10.578>taking <00:09:10.811>these <00:09:11.045>two <00:09:11.278>ideas, <00:09:11.511>we <00:09:11.745>realized

196
00:09:13.235 --> 00:09:16.965
<00:09:13.235>that <00:09:13.390>we <00:09:13.546>could <00:09:13.701>actually <00:09:13.857>get <00:09:14.012>rid <00:09:14.168>of <00:09:14.323>the <00:09:14.479>private <00:09:14.635>databases

197
00:09:17.105 --> 00:09:21.245
<00:09:17.105>and <00:09:17.245>the <00:09:17.385>queries <00:09:17.525>if <00:09:17.665>we <00:09:17.805>simply <00:09:17.945>made <00:09:18.085>sure <00:09:18.225>that <00:09:18.365>all <00:09:18.505>data,

198
00:09:21.545 --> 00:09:23.245
<00:09:21.545>all <00:09:21.895>business <00:09:22.245>data <00:09:22.594>is <00:09:22.944>published

199
00:09:23.305 --> 00:09:25.365
<00:09:23.305>as <00:09:23.584>event <00:09:23.864>streams <00:09:24.145>to <00:09:24.425>Kafka <00:09:24.704>topics.

200
00:09:25.915 --> 00:09:29.085
<00:09:25.915>Because <00:09:26.042>what <00:09:26.169>the <00:09:26.296>services <00:09:26.424>can <00:09:26.551>now <00:09:26.678>do <00:09:26.805>is <00:09:26.933>they <00:09:27.060>can <00:09:27.187>just <00:09:27.314>read

201
00:09:29.905 --> 00:09:31.045
<00:09:29.905>the <00:09:30.161>data <00:09:30.418>that <00:09:30.674>the <00:09:30.930>interest

202
00:09:31.105 --> 00:09:34.365
<00:09:31.105>and <00:09:31.305>write <00:09:31.505>it <00:09:31.705>to <00:09:31.905>some <00:09:32.105>local <00:09:32.305>materialized <00:09:32.505>view

203
00:09:34.365 --> 00:09:37.205
<00:09:34.365>and <00:09:34.565>query <00:09:34.765>that <00:09:34.965>locally <00:09:35.165>instead <00:09:35.365>of <00:09:35.565>having <00:09:35.765>to

204
00:09:37.815 --> 00:09:39.485
<00:09:37.815>query <00:09:38.281>any <00:09:38.748>specific <00:09:39.215>service.

205
00:09:40.265 --> 00:09:42.325
<00:09:40.265>So <00:09:40.420>you <00:09:40.576>don't <00:09:40.731>need <00:09:40.887>the <00:09:41.042>other <00:09:41.198>services <00:09:41.353>to <00:09:41.509>be <00:09:41.665>up.

206
00:09:44.465 --> 00:09:49.365
<00:09:44.465>And, <00:09:44.592>um, <00:09:44.719>yeah, <00:09:44.846>so <00:09:44.974>this <00:09:45.101>is <00:09:45.228>the <00:09:45.355>design <00:09:45.483>we <00:09:45.610>we <00:09:45.737>went <00:09:45.865>with

207
00:09:49.745 --> 00:09:52.125
<00:09:49.745>and, <00:09:49.945>um, <00:09:50.145>of <00:09:50.345>course <00:09:50.545>not <00:09:50.745>with <00:09:50.945>those <00:09:51.145>downsides,

208
00:09:52.145 --> 00:09:54.485
<00:09:52.145>but <00:09:52.285>we'll <00:09:52.425>get <00:09:52.564>to <00:09:52.704>that <00:09:52.845>later <00:09:52.985>in <00:09:53.125>the <00:09:53.265>talk <00:09:53.405>for <00:09:53.545>now.

209
00:09:54.595 --> 00:09:57.405
<00:09:54.595>Back <00:09:54.795>to <00:09:54.995>Ollie <00:09:55.195>for <00:09:55.395>a <00:09:55.595>closer <00:09:55.795>look <00:09:55.995>into

210
00:09:57.665 --> 00:09:59.085
<00:09:57.665>how <00:09:57.920>the <00:09:58.176>system <00:09:58.431>works <00:09:58.687>in <00:09:58.943>detail.

211
00:09:59.395 --> 00:10:01.365
<00:09:59.395>Alright, <00:09:59.595>so <00:09:59.795>we <00:09:59.995>can <00:10:00.194>take <00:10:00.395>that, <00:10:00.595>uh, <00:10:00.795>diagram

212
00:10:01.555 --> 00:10:02.965
<00:10:01.555>that <00:10:01.808>we <00:10:02.062>just <00:10:02.316>saw <00:10:02.570>from <00:10:02.824>Dennis

213
00:10:03.265 --> 00:10:05.445
<00:10:03.265>and <00:10:03.405>look <00:10:03.545>at <00:10:03.685>it <00:10:03.824>from <00:10:03.965>a <00:10:04.105>streams <00:10:04.245>topology <00:10:04.385>point <00:10:04.525>of <00:10:04.665>view.

214
00:10:06.575 --> 00:10:10.405
<00:10:06.575>First <00:10:06.715>of <00:10:06.855>all, <00:10:06.995>we <00:10:07.135>have <00:10:07.275>clients <00:10:07.415>that <00:10:07.555>communicate <00:10:07.695>via, <00:10:07.835>via <00:10:07.975>rest

215
00:10:10.435 --> 00:10:11.845
<00:10:10.435>with <00:10:10.858>an <00:10:11.281>API <00:10:11.704>service.

216
00:10:12.545 --> 00:10:16.325
<00:10:12.545>And <00:10:12.685>this <00:10:12.824>API <00:10:12.964>service <00:10:13.104>has <00:10:13.245>materialized <00:10:13.385>views <00:10:13.525>of <00:10:13.665>all <00:10:13.805>the <00:10:13.944>data

217
00:10:16.635 --> 00:10:18.965
<00:10:16.635>that <00:10:16.835>the <00:10:17.035>client <00:10:17.235>may <00:10:17.435>request <00:10:17.635>from <00:10:17.835>the <00:10:18.035>system.

218
00:10:19.905 --> 00:10:23.765
<00:10:19.905>Um, <00:10:20.032>so <00:10:20.159>if <00:10:20.286>we <00:10:20.414>take <00:10:20.541>the <00:10:20.668>example <00:10:20.795>of <00:10:20.923>an <00:10:21.050>inventory, <00:10:21.177>let's <00:10:21.305>say,

219
00:10:24.505 --> 00:10:27.565
<00:10:24.505>uh, <00:10:24.705>this, <00:10:24.905>these <00:10:25.105>materialized <00:10:25.305>views <00:10:25.505>would <00:10:25.705>contain <00:10:25.905>all

220
00:10:27.685 --> 00:10:31.405
<00:10:27.685>of <00:10:27.824>the <00:10:27.964>inventories <00:10:28.104>and <00:10:28.244>the <00:10:28.385>items <00:10:28.525>that <00:10:28.665>a <00:10:28.805>user, <00:10:28.944>uh, <00:10:29.084>has.

221
00:10:31.585 --> 00:10:34.485
<00:10:31.585>So <00:10:31.740>they <00:10:31.896>can <00:10:32.051>request <00:10:32.207>the <00:10:32.362>contents <00:10:32.518>of <00:10:32.673>their <00:10:32.829>inventory <00:10:32.985>just

222
00:10:34.505 --> 00:10:36.045
<00:10:34.505>by <00:10:34.782>going <00:10:35.059>to <00:10:35.336>the <00:10:35.613>materialized <00:10:35.891>views

223
00:10:36.045 --> 00:10:37.165
<00:10:36.045>and <00:10:36.381>returning <00:10:36.717>those <00:10:37.053>results.

224
00:10:38.185 --> 00:10:43.125
<00:10:38.185>If, <00:10:38.312>uh, <00:10:38.439>a <00:10:38.566>user <00:10:38.694>wants <00:10:38.821>to <00:10:38.948>add <00:10:39.075>an <00:10:39.203>item, <00:10:39.330>then <00:10:39.457>it <00:10:39.584>goes

225
00:10:43.185 --> 00:10:46.085
<00:10:43.185>to <00:10:43.464>a, <00:10:43.744>uh, <00:10:44.025>command <00:10:44.305>processing <00:10:44.584>topology.

226
00:10:46.385 --> 00:10:50.965
<00:10:46.385>So <00:10:46.512>the <00:10:46.639>API <00:10:46.766>writes <00:10:46.894>this <00:10:47.021>ad <00:10:47.148>item <00:10:47.275>command <00:10:47.403>into <00:10:47.530>a <00:10:47.657>command <00:10:47.785>topic,

227
00:10:51.385 --> 00:10:52.445
<00:10:51.385>it <00:10:51.862>gets <00:10:52.339>processed,

228
00:10:52.585 --> 00:10:55.085
<00:10:52.585>and <00:10:52.785>if <00:10:52.985>it's <00:10:53.185>successful, <00:10:53.385>it <00:10:53.585>writes <00:10:53.785>the <00:10:53.985>item

229
00:10:55.265 --> 00:10:57.085
<00:10:55.265>to <00:10:55.545>the <00:10:55.824>inventory <00:10:56.105>at <00:10:56.385>the <00:10:56.665>end.

230
00:10:57.745 --> 00:11:01.605
<00:10:57.745>Uh, <00:10:57.920>the <00:10:58.095>API <00:10:58.270>is <00:10:58.445>then <00:10:58.620>notified <00:10:58.795>of <00:10:58.970>this <00:10:59.145>addition

231
00:11:01.705 --> 00:11:03.525
<00:11:01.705>and <00:11:01.905>can <00:11:02.105>return <00:11:02.305>the <00:11:02.505>results <00:11:02.705>to <00:11:02.905>the <00:11:03.105>client.

232
00:11:05.635 --> 00:11:09.725
<00:11:05.635>Furthermore, <00:11:05.868>we <00:11:06.101>can <00:11:06.335>have <00:11:06.568>multiple <00:11:06.801>downstream <00:11:07.035>apologies

233
00:11:10.275 --> 00:11:13.685
<00:11:10.275>that <00:11:10.449>then <00:11:10.625>start <00:11:10.800>processing <00:11:10.975>these <00:11:11.150>changes <00:11:11.324>in <00:11:11.500>the <00:11:11.675>inventory.

234
00:11:14.145 --> 00:11:16.645
<00:11:14.145>So <00:11:14.319>we <00:11:14.495>could <00:11:14.670>have <00:11:14.845>a <00:11:15.020>downstream <00:11:15.194>service, <00:11:15.370>for <00:11:15.545>example,

235
00:11:16.645 --> 00:11:18.965
<00:11:16.645>which <00:11:16.995>enriches <00:11:17.345>data <00:11:17.694>of <00:11:18.045>inventories

236
00:11:18.985 --> 00:11:22.685
<00:11:18.985>and <00:11:19.160>writes <00:11:19.335>those <00:11:19.510>results <00:11:19.685>to <00:11:19.860>yet <00:11:20.035>further <00:11:20.210>downstream <00:11:20.385>topics.

237
00:11:24.705 --> 00:11:27.165
<00:11:24.705>So <00:11:24.880>what <00:11:25.055>do <00:11:25.230>the <00:11:25.405>failure <00:11:25.580>scenarios <00:11:25.755>here <00:11:25.930>look <00:11:26.105>like?

238
00:11:27.665 --> 00:11:30.965
<00:11:27.665>Uh, <00:11:27.792>how <00:11:27.919>can, <00:11:28.046>what <00:11:28.174>does <00:11:28.301>this <00:11:28.428>mean <00:11:28.555>when <00:11:28.683>we <00:11:28.810>talk <00:11:28.937>about <00:11:29.064>failure

239
00:11:31.115 --> 00:11:32.445
<00:11:31.115>tolerance <00:11:31.713>and <00:11:32.312>robustness?

240
00:11:33.705 --> 00:11:35.885
<00:11:33.705>So <00:11:33.985>one <00:11:34.265>of <00:11:34.545>the <00:11:34.825>cases, <00:11:35.105>uh,

241
00:11:36.025 --> 00:11:38.525
<00:11:36.025>we <00:11:36.225>could <00:11:36.425>have <00:11:36.625>buggy <00:11:36.824>logic <00:11:37.025>in <00:11:37.225>our <00:11:37.425>microservice.

242
00:11:39.305 --> 00:11:43.965
<00:11:39.305>So <00:11:39.460>it <00:11:39.616>is <00:11:39.771>writing <00:11:39.927>invalid <00:11:40.082>data <00:11:40.238>into <00:11:40.393>a <00:11:40.549>downstream <00:11:40.704>topic.

243
00:11:44.065 --> 00:11:46.285
<00:11:44.065>How <00:11:44.265>do <00:11:44.465>we <00:11:44.665>fix <00:11:44.865>it? <00:11:45.065>We <00:11:45.265>fix <00:11:45.465>it

244
00:11:46.345 --> 00:11:49.805
<00:11:46.345>by <00:11:46.520>simply <00:11:46.695>deploying <00:11:46.870>a <00:11:47.045>fixed <00:11:47.220>version <00:11:47.395>of <00:11:47.570>that, <00:11:47.745>uh,

245
00:11:49.805 --> 00:11:53.005
<00:11:49.805>microservice <00:11:50.038>and <00:11:50.271>then <00:11:50.505>make <00:11:50.738>it <00:11:50.971>replay <00:11:51.204>all

246
00:11:53.085 --> 00:11:54.525
<00:11:53.085>of <00:11:53.301>the <00:11:53.517>data <00:11:53.733>in <00:11:53.949>its <00:11:54.165>input <00:11:54.381>topics

247
00:11:55.025 --> 00:11:56.925
<00:11:55.025>and <00:11:55.305>then <00:11:55.584>write <00:11:55.865>that <00:11:56.145>data, <00:11:56.425>uh,

248
00:11:57.065 --> 00:11:59.245
<00:11:57.065>as <00:11:57.345>fixed <00:11:57.625>into <00:11:57.905>its <00:11:58.185>output <00:11:58.465>topic.

249
00:11:59.585 --> 00:12:03.485
<00:11:59.585>So <00:11:59.712>now <00:11:59.839>we <00:11:59.966>are <00:12:00.094>able <00:12:00.221>to <00:12:00.348>fix <00:12:00.475>invalid <00:12:00.603>data <00:12:00.730>in <00:12:00.857>output <00:12:00.985>topics.

250
00:12:04.825 --> 00:12:07.685
<00:12:04.825>We <00:12:05.000>can <00:12:05.175>also <00:12:05.350>have <00:12:05.525>a <00:12:05.700>microservice <00:12:05.875>go <00:12:06.050>down <00:12:06.225>indefinitely

251
00:12:08.075 --> 00:12:10.885
<00:12:08.075>because <00:12:08.308>we <00:12:08.541>keep <00:12:08.775>the <00:12:09.008>input <00:12:09.241>data <00:12:09.475>indefinitely.

252
00:12:11.665 --> 00:12:13.445
<00:12:11.665>If <00:12:11.944>it <00:12:12.224>comes <00:12:12.505>back <00:12:12.785>up <00:12:13.064>later,

253
00:12:13.905 --> 00:12:16.165
<00:12:13.905>it <00:12:14.105>continues <00:12:14.305>reading <00:12:14.505>from <00:12:14.704>where <00:12:14.905>it <00:12:15.105>left <00:12:15.305>off.

254
00:12:16.705 --> 00:12:20.245
<00:12:16.705>Uh, <00:12:16.880>and, <00:12:17.055>uh, <00:12:17.230>there <00:12:17.405>is <00:12:17.580>no <00:12:17.755>data <00:12:17.930>consistency <00:12:18.105>issue.

255
00:12:20.355 --> 00:12:22.045
<00:12:20.355>It's <00:12:21.055>eventually <00:12:21.755>consistent.

256
00:12:23.705 --> 00:12:26.605
<00:12:23.705>The <00:12:23.845>third <00:12:23.985>failure <00:12:24.125>scenario <00:12:24.265>is <00:12:24.405>if <00:12:24.545>we <00:12:24.685>have <00:12:24.825>a <00:12:24.965>momentary <00:12:25.105>glitch

257
00:12:26.605 --> 00:12:29.765
<00:12:26.605>in <00:12:26.805>the <00:12:27.005>command <00:12:27.205>processing <00:12:27.405>topology, <00:12:27.605>for <00:12:27.805>example, <00:12:28.005>uh,

258
00:12:29.865 --> 00:12:32.965
<00:12:29.865>in <00:12:30.065>this <00:12:30.265>case, <00:12:30.465>similar <00:12:30.665>to <00:12:30.865>message <00:12:31.065>queues, <00:12:31.265>uh,

259
00:12:33.025 --> 00:12:36.925
<00:12:33.025>the <00:12:33.180>end <00:12:33.336>user <00:12:33.491>will <00:12:33.647>just <00:12:33.802>see <00:12:33.958>an <00:12:34.113>increased <00:12:34.269>latency, <00:12:34.425>uh,

260
00:12:36.925 --> 00:12:38.405
<00:12:36.925>when <00:12:37.257>this <00:12:37.591>momentary <00:12:37.924>glitch <00:12:38.257>happens.

261
00:12:38.505 --> 00:12:40.045
<00:12:38.505>If <00:12:38.851>it <00:12:39.198>recovers <00:12:39.544>quickly <00:12:39.891>enough.

262
00:12:41.625 --> 00:12:43.805
<00:12:41.625>The <00:12:41.780>fourth <00:12:41.936>case, <00:12:42.091>I <00:12:42.247>think <00:12:42.402>is <00:12:42.558>the <00:12:42.713>most <00:12:42.869>interesting <00:12:43.025>one.

263
00:12:44.145 --> 00:12:47.645
<00:12:44.145>So <00:12:44.272>what <00:12:44.399>happens <00:12:44.526>if <00:12:44.654>the <00:12:44.781>API <00:12:44.908>service <00:12:45.035>gets <00:12:45.163>cut <00:12:45.290>off <00:12:45.417>from <00:12:45.545>all

264
00:12:47.645 --> 00:12:50.165
<00:12:47.645>of <00:12:47.785>the <00:12:47.925>rest <00:12:48.064>of <00:12:48.204>the <00:12:48.345>system, <00:12:48.485>all <00:12:48.625>of <00:12:48.765>the <00:12:48.905>sub, <00:12:49.045>all

265
00:12:50.165 --> 00:12:52.205
<00:12:50.165>of <00:12:50.398>the <00:12:50.631>topologies <00:12:50.865>and <00:12:51.098>the <00:12:51.331>other <00:12:51.564>microservices?

266
00:12:53.115 --> 00:12:57.325
<00:12:53.115>It's <00:12:53.270>actually <00:12:53.426>okay <00:12:53.581>because <00:12:53.737>well, <00:12:53.892>it <00:12:54.048>can <00:12:54.203>no <00:12:54.359>longer, <00:12:54.515>uh,

267
00:12:57.355 --> 00:12:58.365
<00:12:57.355>execute <00:12:58.264>commands,

268
00:12:58.825 --> 00:13:02.045
<00:12:58.825>but <00:12:58.952>it <00:12:59.079>can <00:12:59.206>still <00:12:59.334>provide <00:12:59.461>all <00:12:59.588>of <00:12:59.715>the <00:12:59.843>data <00:12:59.970>to <00:13:00.097>the <00:13:00.225>clients.

269
00:13:02.185 --> 00:13:06.765
<00:13:02.185>So <00:13:02.340>data <00:13:02.496>availability <00:13:02.651>does <00:13:02.807>not <00:13:02.962>actually <00:13:03.118>depend <00:13:03.273>on <00:13:03.429>the <00:13:03.584>uptime

270
00:13:06.765 --> 00:13:10.565
<00:13:06.765>of <00:13:06.920>the <00:13:07.076>rest <00:13:07.231>of <00:13:07.387>the <00:13:07.542>system <00:13:07.698>making <00:13:07.853>reads <00:13:08.009>extremely <00:13:08.165>robust.

271
00:13:11.985 --> 00:13:13.965
<00:13:11.985>All <00:13:12.140>right, <00:13:12.296>let's <00:13:12.451>have <00:13:12.607>a <00:13:12.762>look <00:13:12.918>at <00:13:13.073>some <00:13:13.229>real <00:13:13.385>life

272
00:13:14.205 --> 00:13:15.685
<00:13:14.205>examples <00:13:14.649>of <00:13:15.093>the <00:13:15.536>pipelines.

273
00:13:16.225 --> 00:13:17.885
<00:13:16.225>The <00:13:16.458>first <00:13:16.691>one <00:13:16.925>is <00:13:17.158>the <00:13:17.391>product <00:13:17.625>catalog.

274
00:13:18.825 --> 00:13:21.325
<00:13:18.825>So <00:13:18.941>what <00:13:19.058>we <00:13:19.175>needed <00:13:19.291>to <00:13:19.408>do <00:13:19.525>was <00:13:19.641>we <00:13:19.758>needed <00:13:19.875>to <00:13:19.991>get <00:13:20.108>a <00:13:20.225>source

275
00:13:21.325 --> 00:13:24.765
<00:13:21.325>of <00:13:21.525>product <00:13:21.725>data <00:13:21.925>into <00:13:22.125>our <00:13:22.325>platform, <00:13:22.525>uh, <00:13:22.725>so

276
00:13:24.765 --> 00:13:26.845
<00:13:24.765>that <00:13:24.965>it <00:13:25.165>can <00:13:25.365>be <00:13:25.564>consumed <00:13:25.765>by <00:13:25.965>Kafka <00:13:26.165>streams.

277
00:13:27.865 --> 00:13:29.765
<00:13:27.865>We <00:13:28.098>also <00:13:28.331>needed <00:13:28.565>a <00:13:28.798>way <00:13:29.031>to <00:13:29.265>amend

278
00:13:29.765 --> 00:13:33.005
<00:13:29.765>and <00:13:29.905>correct <00:13:30.045>data <00:13:30.185>in <00:13:30.324>this <00:13:30.465>catalog <00:13:30.605>and <00:13:30.745>to <00:13:30.885>make <00:13:31.025>it <00:13:31.165>searchable.

279
00:13:33.705 --> 00:13:37.125
<00:13:33.705>So <00:13:33.860>how <00:13:34.016>do <00:13:34.171>we <00:13:34.327>do <00:13:34.482>it? <00:13:34.638>First <00:13:34.793>we <00:13:34.949>have <00:13:35.105>catalogs.

280
00:13:37.985 --> 00:13:40.285
<00:13:37.985>We <00:13:38.218>then <00:13:38.451>parse <00:13:38.685>them <00:13:38.918>using <00:13:39.151>a <00:13:39.385>parser

281
00:13:40.285 --> 00:13:43.245
<00:13:40.285>that <00:13:40.485>we <00:13:40.685>built <00:13:40.885>into <00:13:41.084>a <00:13:41.285>parsed <00:13:41.485>products <00:13:41.685>topic.

282
00:13:43.705 --> 00:13:45.565
<00:13:43.705>Now, <00:13:43.938>the <00:13:44.171>data <00:13:44.405>here <00:13:44.638>is <00:13:44.871>quite <00:13:45.105>unclean.

283
00:13:45.865 --> 00:13:48.285
<00:13:45.865>Uh, <00:13:46.065>there <00:13:46.265>are <00:13:46.465>a <00:13:46.665>lot <00:13:46.865>of <00:13:47.065>duplicates, <00:13:47.265>uh,

284
00:13:48.285 --> 00:13:50.245
<00:13:48.285>from <00:13:48.485>all <00:13:48.685>the <00:13:48.885>different <00:13:49.084>versions <00:13:49.285>of <00:13:49.485>the <00:13:49.685>catalog.

285
00:13:50.815 --> 00:13:55.205
<00:13:50.815>There <00:13:50.970>might <00:13:51.126>also <00:13:51.281>be <00:13:51.437>some, <00:13:51.592>uh, <00:13:51.748>faulty <00:13:51.903>data, <00:13:52.059>invalid <00:13:52.215>data,

286
00:13:55.315 --> 00:13:56.885
<00:13:55.315>missing <00:13:55.665>data <00:13:56.015>and <00:13:56.365>so <00:13:56.715>on.

287
00:13:57.385 --> 00:14:00.685
<00:13:57.385>So <00:13:57.512>how <00:13:57.639>do <00:13:57.766>we <00:13:57.894>clean <00:13:58.021>it? <00:13:58.148>We <00:13:58.275>clean <00:13:58.403>it <00:13:58.530>by <00:13:58.657>attaching <00:13:58.785>it

288
00:14:00.685 --> 00:14:04.125
<00:14:00.685>to <00:14:00.918>a <00:14:01.151>product <00:14:01.385>processor, <00:14:01.618>which <00:14:01.851>for <00:14:02.084>example,

289
00:14:04.305 --> 00:14:06.165
<00:14:04.305>can <00:14:04.771>remove <00:14:05.238>duplicates, <00:14:05.704>uh,

290
00:14:06.185 --> 00:14:08.325
<00:14:06.185>and <00:14:06.464>choose <00:14:06.744>the <00:14:07.025>most <00:14:07.305>relevant <00:14:07.584>version

291
00:14:08.385 --> 00:14:11.845
<00:14:08.385>of <00:14:08.525>a <00:14:08.665>given <00:14:08.805>product <00:14:08.944>model, <00:14:09.085>uh, <00:14:09.225>from <00:14:09.365>all <00:14:09.505>of <00:14:09.645>those <00:14:09.785>catalogs,

292
00:14:12.065 --> 00:14:15.125
<00:14:12.065>and <00:14:12.220>then <00:14:12.376>writes <00:14:12.531>that <00:14:12.687>best <00:14:12.842>version <00:14:12.998>into <00:14:13.153>the <00:14:13.309>products <00:14:13.465>topic.

293
00:14:16.025 --> 00:14:18.885
<00:14:16.025>The <00:14:16.199>Kafka <00:14:16.375>streams <00:14:16.550>code <00:14:16.725>is <00:14:16.900>actually <00:14:17.074>very <00:14:17.250>simple <00:14:17.425>here.

294
00:14:19.035 --> 00:14:20.165
<00:14:19.035>This <00:14:19.374>is <00:14:19.713>basically <00:14:20.051>it.

295
00:14:20.865 --> 00:14:22.525
<00:14:20.865>You <00:14:21.565>group <00:14:22.265>all

296
00:14:22.525 --> 00:14:24.485
<00:14:22.525>of <00:14:22.699>the <00:14:22.875>products <00:14:23.050>coming <00:14:23.225>in <00:14:23.400>from <00:14:23.574>the <00:14:23.750>pars <00:14:23.925>products

297
00:14:24.625 --> 00:14:26.885
<00:14:24.625>by <00:14:24.780>their <00:14:24.936>key, <00:14:25.091>which <00:14:25.247>is <00:14:25.402>like <00:14:25.558>the <00:14:25.713>product <00:14:25.869>model <00:14:26.025>id.

298
00:14:27.705 --> 00:14:30.005
<00:14:27.705>And <00:14:27.985>then <00:14:28.265>you <00:14:28.545>reduce <00:14:28.825>this <00:14:29.105>aggregation

299
00:14:30.185 --> 00:14:32.165
<00:14:30.185>by <00:14:30.464>choosing <00:14:30.744>the <00:14:31.025>most <00:14:31.305>relevant <00:14:31.584>product.

300
00:14:32.625 --> 00:14:35.445
<00:14:32.625>And <00:14:32.825>relevance <00:14:33.025>here <00:14:33.225>can <00:14:33.425>be <00:14:33.625>defined, <00:14:33.825>for <00:14:34.025>example,

301
00:14:35.705 --> 00:14:37.245
<00:14:35.705>by <00:14:36.051>the <00:14:36.398>last <00:14:36.744>modified <00:14:37.091>date,

302
00:14:37.425 --> 00:14:40.485
<00:14:37.425>or <00:14:37.580>it <00:14:37.736>can <00:14:37.891>also <00:14:38.047>be, <00:14:38.202>uh, <00:14:38.358>based <00:14:38.513>on <00:14:38.669>a <00:14:38.824>decision

303
00:14:40.485 --> 00:14:41.845
<00:14:40.485>of <00:14:40.791>looking <00:14:41.097>at <00:14:41.403>the <00:14:41.709>properties,

304
00:14:41.845 --> 00:14:43.445
<00:14:41.845>whether <00:14:42.195>the <00:14:42.545>most <00:14:42.895>important <00:14:43.245>pieces

305
00:14:43.465 --> 00:14:44.965
<00:14:43.465>of <00:14:43.802>information <00:14:44.140>are <00:14:44.477>actually <00:14:44.815>present.

306
00:14:45.945 --> 00:14:48.485
<00:14:45.945>And <00:14:46.120>then <00:14:46.295>we <00:14:46.470>simply <00:14:46.645>convert <00:14:46.820>it <00:14:46.995>into <00:14:47.170>a <00:14:47.345>stream

307
00:14:48.665 --> 00:14:50.725
<00:14:48.665>and <00:14:48.865>stream <00:14:49.064>it <00:14:49.265>out <00:14:49.464>into <00:14:49.665>a <00:14:49.865>products <00:14:50.064>topic.

308
00:14:52.465 --> 00:14:53.645
<00:14:52.465>Now <00:14:52.730>we <00:14:52.996>have <00:14:53.261>products <00:14:53.527>data

309
00:14:53.715 --> 00:14:57.725
<00:14:53.715>that <00:14:53.855>can <00:14:53.995>be <00:14:54.135>joined <00:14:54.275>in <00:14:54.415>the <00:14:54.555>platform, <00:14:54.695>uh, <00:14:54.835>in <00:14:54.975>other <00:14:55.115>streams,

310
00:14:58.265 --> 00:15:00.605
<00:14:58.265>and <00:14:58.498>we <00:14:58.731>can <00:14:58.965>have <00:14:59.198>many, <00:14:59.431>many, <00:14:59.665>uh,

311
00:15:00.605 --> 00:15:02.565
<00:15:00.605>different <00:15:00.955>streams <00:15:01.305>joining <00:15:01.655>this <00:15:02.005>information.

312
00:15:02.945 --> 00:15:04.165
<00:15:02.945>And <00:15:03.219>it's <00:15:03.494>actually <00:15:03.768>quite <00:15:04.043>scalable

313
00:15:04.165 --> 00:15:07.725
<00:15:04.165>because <00:15:04.365>there's <00:15:04.564>no <00:15:04.765>service <00:15:04.964>that <00:15:05.165>needs <00:15:05.365>to, <00:15:05.564>uh,

314
00:15:08.115 --> 00:15:10.085
<00:15:08.115>provide <00:15:08.581>these <00:15:09.048>products, <00:15:09.515>uh,

315
00:15:10.085 --> 00:15:11.605
<00:15:10.085>this <00:15:10.427>product <00:15:10.769>data <00:15:11.111>through <00:15:11.453>queries.

316
00:15:12.945 --> 00:15:14.685
<00:15:12.945>But <00:15:13.178>now <00:15:13.411>we <00:15:13.645>have <00:15:13.878>a <00:15:14.111>new <00:15:14.345>requirement.

317
00:15:14.985 --> 00:15:16.805
<00:15:14.985>Um, <00:15:15.265>we <00:15:15.545>have <00:15:15.825>some <00:15:16.105>missing <00:15:16.385>data,

318
00:15:16.865 --> 00:15:18.085
<00:15:16.865>for <00:15:17.084>example, <00:15:17.304>in <00:15:17.523>the <00:15:17.743>products <00:15:17.963>topic.

319
00:15:18.905 --> 00:15:23.445
<00:15:18.905>So <00:15:19.021>we <00:15:19.138>need <00:15:19.255>to <00:15:19.371>be <00:15:19.488>able <00:15:19.605>to <00:15:19.721>read <00:15:19.838>a <00:15:19.954>source <00:15:20.071>of <00:15:20.188>amended <00:15:20.305>products,

320
00:15:23.575 --> 00:15:26.285
<00:15:23.575>which <00:15:23.808>is <00:15:24.041>modified <00:15:24.275>by <00:15:24.508>back <00:15:24.741>office <00:15:24.975>employees.

321
00:15:26.285 --> 00:15:28.645
<00:15:26.285>So <00:15:26.485>this <00:15:26.685>is <00:15:26.885>like <00:15:27.084>a <00:15:27.285>classical <00:15:27.485>relational <00:15:27.685>database.

322
00:15:29.385 --> 00:15:31.765
<00:15:29.385>We <00:15:29.585>then <00:15:29.785>deploy <00:15:29.985>a <00:15:30.185>technology <00:15:30.385>called <00:15:30.585>Kafka <00:15:30.785>Connect

323
00:15:32.065 --> 00:15:35.445
<00:15:32.065>to <00:15:32.192>stream <00:15:32.319>all <00:15:32.446>of <00:15:32.574>the <00:15:32.701>data <00:15:32.828>in <00:15:32.955>the <00:15:33.083>amended <00:15:33.210>products <00:15:33.337>topic, <00:15:33.465>uh,

324
00:15:35.445 --> 00:15:37.285
<00:15:35.445>sorry, <00:15:35.725>in <00:15:36.005>the <00:15:36.285>relational <00:15:36.565>database <00:15:36.845>into

325
00:15:37.285 --> 00:15:38.405
<00:15:37.285>an <00:15:37.621>amended <00:15:37.957>products <00:15:38.293>topic.

326
00:15:38.985 --> 00:15:41.445
<00:15:38.985>And <00:15:39.265>then <00:15:39.545>we <00:15:39.825>simply <00:15:40.105>concatenate <00:15:40.385>this

327
00:15:41.915 --> 00:15:45.365
<00:15:41.915>with <00:15:42.089>the <00:15:42.265>past <00:15:42.439>products <00:15:42.615>topic <00:15:42.790>in <00:15:42.964>the <00:15:43.140>product <00:15:43.314>processor

328
00:15:45.665 --> 00:15:50.085
<00:15:45.665>and <00:15:45.820>make <00:15:45.976>a <00:15:46.131>slight <00:15:46.287>modification <00:15:46.442>in <00:15:46.598>the <00:15:46.753>relevance <00:15:46.909>code, <00:15:47.064>uh,

329
00:15:50.145 --> 00:15:52.805
<00:15:50.145>to <00:15:50.345>select <00:15:50.545>the <00:15:50.745>amended <00:15:50.944>product <00:15:51.145>if <00:15:51.345>it <00:15:51.545>exists.

330
00:15:53.505 --> 00:15:57.325
<00:15:53.505>And <00:15:53.645>now <00:15:53.785>we <00:15:53.925>have <00:15:54.064>the <00:15:54.205>capability <00:15:54.345>to <00:15:54.485>fix <00:15:54.625>broken <00:15:54.765>products <00:15:54.905>data.

331
00:15:58.465 --> 00:16:00.205
<00:15:58.465>So <00:15:58.698>now <00:15:58.931>we <00:15:59.165>have <00:15:59.398>yet <00:15:59.631>another <00:15:59.865>requirement.

332
00:16:00.305 --> 00:16:02.325
<00:16:00.305>We <00:16:00.479>want <00:16:00.655>to <00:16:00.829>be <00:16:01.005>able <00:16:01.180>to <00:16:01.354>search <00:16:01.530>our <00:16:01.704>products

333
00:16:03.025 --> 00:16:04.725
<00:16:03.025>and, <00:16:03.225>uh, <00:16:03.425>how <00:16:03.625>do <00:16:03.824>we <00:16:04.025>make <00:16:04.225>this <00:16:04.425>happen?

334
00:16:04.845 --> 00:16:06.485
<00:16:04.845>Because <00:16:05.311>in <00:16:05.778>Kafka, <00:16:06.245>uh,

335
00:16:06.545 --> 00:16:09.285
<00:16:06.545>you <00:16:06.685>can <00:16:06.824>only <00:16:06.964>look <00:16:07.104>up <00:16:07.245>data <00:16:07.385>if <00:16:07.525>you <00:16:07.665>know <00:16:07.805>it's <00:16:07.944>key,

336
00:16:09.825 --> 00:16:11.725
<00:16:09.825>but <00:16:09.965>we <00:16:10.105>want <00:16:10.245>to <00:16:10.385>be <00:16:10.525>able <00:16:10.665>to <00:16:10.805>search <00:16:10.945>by, <00:16:11.085>for <00:16:11.225>example,

337
00:16:11.725 --> 00:16:14.765
<00:16:11.725>product <00:16:11.925>name <00:16:12.125>or <00:16:12.325>description <00:16:12.525>to <00:16:12.725>enable <00:16:12.925>this <00:16:13.125>scenario.

338
00:16:15.305 --> 00:16:17.405
<00:16:15.305>We <00:16:15.584>deploy <00:16:15.864>a <00:16:16.145>product <00:16:16.425>search <00:16:16.704>processor,

339
00:16:17.575 --> 00:16:19.525
<00:16:17.575>which <00:16:18.041>simply <00:16:18.508>streams <00:16:18.975>all

340
00:16:19.525 --> 00:16:22.845
<00:16:19.525>of <00:16:19.665>the <00:16:19.805>data <00:16:19.944>from <00:16:20.084>the <00:16:20.225>products <00:16:20.365>topic <00:16:20.505>into <00:16:20.645>search <00:16:20.785>engine <00:16:20.925>such

341
00:16:22.845 --> 00:16:23.885
<00:16:22.845>as <00:16:23.313>elastic <00:16:23.781>Elasticsearch.

342
00:16:24.225 --> 00:16:26.245
<00:16:24.225>So <00:16:24.458>this <00:16:24.691>is <00:16:24.925>now <00:16:25.158>a <00:16:25.391>materialized <00:16:25.625>view.

343
00:16:27.185 --> 00:16:30.205
<00:16:27.185>What's <00:16:27.385>very <00:16:27.584>nice <00:16:27.785>about <00:16:27.984>this <00:16:28.185>is <00:16:28.385>that, <00:16:28.584>uh,

344
00:16:30.465 --> 00:16:33.845
<00:16:30.465>we <00:16:30.605>didn't <00:16:30.745>actually <00:16:30.885>have <00:16:31.025>to <00:16:31.165>consider <00:16:31.305>what <00:16:31.445>sort <00:16:31.585>of <00:16:31.725>a, <00:16:31.865>uh,

345
00:16:34.125 --> 00:16:37.485
<00:16:34.125>database <00:16:34.300>technology <00:16:34.475>we <00:16:34.650>used <00:16:34.825>for <00:16:35.000>these, <00:16:35.175>uh, <00:16:35.350>products <00:16:35.525>data,

346
00:16:38.385 --> 00:16:43.125
<00:16:38.385>um, <00:16:38.560>until <00:16:38.735>we <00:16:38.910>actually <00:16:39.085>had <00:16:39.260>the <00:16:39.435>requirements, <00:16:39.610>uh, <00:16:39.785>for

347
00:16:43.145 --> 00:16:44.245
<00:16:43.145>how <00:16:43.343>we <00:16:43.540>want <00:16:43.739>to <00:16:43.937>query <00:16:44.135>it.

348
00:16:44.465 --> 00:16:46.725
<00:16:44.465>So <00:16:44.620>usually <00:16:44.776>you <00:16:44.931>have <00:16:45.087>to <00:16:45.242>take <00:16:45.398>those <00:16:45.553>querying <00:16:45.709>requirements <00:16:45.865>into

349
00:16:46.885 --> 00:16:48.285
<00:16:46.885>consideration <00:16:47.199>when <00:16:47.515>making <00:16:47.829>those <00:16:48.145>decisions.

350
00:16:48.945 --> 00:16:51.845
<00:16:48.945>Uh, <00:16:49.178>let's <00:16:49.411>say <00:16:49.645>for <00:16:49.878>example <00:16:50.111>tomorrow, <00:16:50.345>uh,

351
00:16:51.845 --> 00:16:53.805
<00:16:51.845>we <00:16:52.000>get <00:16:52.156>a <00:16:52.311>new <00:16:52.467>requirement, <00:16:52.622>which <00:16:52.778>is <00:16:52.933>to <00:16:53.089>be <00:16:53.245>able

352
00:16:53.805 --> 00:16:56.445
<00:16:53.805>to <00:16:54.084>search <00:16:54.364>products <00:16:54.645>through <00:16:54.925>product <00:16:55.204>relationships.

353
00:16:57.065 --> 00:17:01.125
<00:16:57.065>We <00:16:57.220>can <00:16:57.376>simply <00:16:57.531>stream <00:16:57.687>out <00:16:57.842>those <00:16:57.998>products <00:16:58.153>into <00:16:58.309>a <00:16:58.465>graph

354
00:17:01.405 --> 00:17:04.005
<00:17:01.405>database <00:17:01.605>as <00:17:01.805>well <00:17:02.005>and <00:17:02.204>then <00:17:02.405>let <00:17:02.605>clients <00:17:02.805>query

355
00:17:04.035 --> 00:17:06.525
<00:17:04.035>that, <00:17:04.735>right? <00:17:05.435>Alright,

356
00:17:06.525 --> 00:17:07.525
<00:17:06.525>Yeah. <00:17:06.825>Second <00:17:07.125>use <00:17:07.425>case

357
00:17:07.525 --> 00:17:09.765
<00:17:07.525>we <00:17:07.875>implemented <00:17:08.225>is <00:17:08.575>inventory <00:17:08.925>management.

358
00:17:10.265 --> 00:17:13.645
<00:17:10.265>We <00:17:10.420>wanted <00:17:10.576>users <00:17:10.731>to <00:17:10.887>have <00:17:11.042>inventories <00:17:11.198>where <00:17:11.353>they <00:17:11.509>could <00:17:11.665>add

359
00:17:13.705 --> 00:17:15.005
<00:17:13.704>and <00:17:13.997>edit <00:17:14.290>and <00:17:14.582>remove <00:17:14.875>items

360
00:17:15.095 --> 00:17:18.525
<00:17:15.095>where <00:17:15.270>items <00:17:15.444>are <00:17:15.620>usually <00:17:15.795>devices <00:17:15.970>manufactured <00:17:16.145>by <00:17:16.319>our <00:17:16.495>client.

361
00:17:19.705 --> 00:17:22.925
<00:17:19.704>And <00:17:19.860>as <00:17:20.016>another <00:17:20.171>requirement, <00:17:20.327>we <00:17:20.482>wanted <00:17:20.638>to <00:17:20.793>keep <00:17:20.949>the <00:17:21.105>history

362
00:17:22.985 --> 00:17:24.245
<00:17:22.984>of <00:17:23.551>inventory <00:17:24.118>actions.

363
00:17:25.105 --> 00:17:27.845
<00:17:25.105>So <00:17:25.245>it <00:17:25.385>should <00:17:25.525>be <00:17:25.665>possible <00:17:25.805>to, <00:17:25.944>for <00:17:26.085>instance, <00:17:26.224>play <00:17:26.365>some <00:17:26.505>kind

364
00:17:27.845 --> 00:17:30.445
<00:17:27.845>of <00:17:28.020>timeline <00:17:28.194>view <00:17:28.370>of <00:17:28.545>what <00:17:28.720>happened <00:17:28.895>in <00:17:29.070>the <00:17:29.245>inventory

365
00:17:30.545 --> 00:17:32.925
<00:17:30.545>or <00:17:30.745>do <00:17:30.945>auditing <00:17:31.145>on <00:17:31.345>the <00:17:31.545>past <00:17:31.745>inventory <00:17:31.945>actions.

366
00:17:34.185 --> 00:17:37.725
<00:17:34.185>So <00:17:34.340>the <00:17:34.496>pipeline <00:17:34.651>we <00:17:34.807>came <00:17:34.962>up <00:17:35.118>with <00:17:35.273>starts <00:17:35.429>with, <00:17:35.585>uh,

367
00:17:38.045 --> 00:17:42.165
<00:17:38.045>a <00:17:38.172>command <00:17:38.299>coming <00:17:38.426>in <00:17:38.554>from <00:17:38.681>the <00:17:38.808>front <00:17:38.935>end <00:17:39.063>to <00:17:39.190>our <00:17:39.317>rest <00:17:39.445>API,

368
00:17:42.615 --> 00:17:46.845
<00:17:42.615>which <00:17:42.770>is <00:17:42.926>then <00:17:43.081>translated <00:17:43.237>into <00:17:43.392>our <00:17:43.548>commands <00:17:43.703>into <00:17:43.859>a <00:17:44.015>Kafka

369
00:17:46.845 --> 00:17:48.245
<00:17:46.845>topic, <00:17:47.264>inventory <00:17:47.685>commands <00:17:48.105>topic.

370
00:17:49.395 --> 00:17:52.165
<00:17:49.395>This <00:17:49.595>is <00:17:49.795>then <00:17:49.995>read <00:17:50.195>by <00:17:50.395>an <00:17:50.595>command <00:17:50.795>processor,

371
00:17:52.205 --> 00:17:55.005
<00:17:52.205>a <00:17:52.379>Kafka <00:17:52.554>streams <00:17:52.730>processor, <00:17:52.905>which <00:17:53.080>then <00:17:53.254>validates <00:17:53.429>the <00:17:53.605>command.

372
00:17:55.345 --> 00:17:58.605
<00:17:55.345>And <00:17:55.461>then <00:17:55.578>if <00:17:55.695>everything <00:17:55.811>is <00:17:55.928>fine, <00:17:56.045>it <00:17:56.161>writes <00:17:56.278>it <00:17:56.395>out <00:17:56.511>as <00:17:56.628>an <00:17:56.745>event

373
00:17:58.905 --> 00:18:01.285
<00:17:58.905>to <00:17:59.185>an <00:17:59.465>inventory <00:17:59.745>item <00:18:00.024>Event <00:18:00.305>topic.

374
00:18:01.435 --> 00:18:03.485
<00:18:01.435>This <00:18:01.590>now <00:18:01.746>has <00:18:01.901>creation <00:18:02.057>and <00:18:02.212>update <00:18:02.368>in <00:18:02.523>the <00:18:02.679>lesion <00:18:02.835>events.

375
00:18:04.625 --> 00:18:07.925
<00:18:04.625>And <00:18:04.800>this <00:18:04.975>topic <00:18:05.150>we <00:18:05.325>consider <00:18:05.500>the <00:18:05.675>source <00:18:05.850>of <00:18:06.025>truth

376
00:18:08.385 --> 00:18:09.885
<00:18:08.385>for <00:18:09.060>inventory <00:18:09.735>data,

377
00:18:10.745 --> 00:18:13.965
<00:18:10.745>but <00:18:10.919>for <00:18:11.094>convenience <00:18:11.270>we <00:18:11.445>also <00:18:11.620>have <00:18:11.794>a <00:18:11.969>reduction <00:18:12.145>step

378
00:18:14.095 --> 00:18:15.485
<00:18:14.095>where <00:18:14.407>we <00:18:14.720>take <00:18:15.033>these <00:18:15.346>events

379
00:18:15.705 --> 00:18:18.685
<00:18:15.705>and <00:18:15.860>reduce <00:18:16.016>them <00:18:16.171>to <00:18:16.327>the <00:18:16.482>items <00:18:16.638>as <00:18:16.793>they <00:18:16.949>currently <00:18:17.105>are,

380
00:18:19.175 --> 00:18:22.045
<00:18:19.175>which <00:18:19.330>makes <00:18:19.486>it <00:18:19.641>a <00:18:19.797>bit <00:18:19.952>easier <00:18:20.108>to <00:18:20.263>create <00:18:20.419>materialized <00:18:20.575>views.

381
00:18:22.435 --> 00:18:25.765
<00:18:22.435>Like <00:18:22.575>for <00:18:22.715>instance, <00:18:22.855>the <00:18:22.995>rest <00:18:23.135>API <00:18:23.274>does <00:18:23.415>to <00:18:23.554>serve <00:18:23.695>with <00:18:23.835>requests.

382
00:18:27.065 --> 00:18:30.165
<00:18:27.065>So <00:18:27.240>this <00:18:27.415>pipeline <00:18:27.590>worked <00:18:27.765>for <00:18:27.940>the <00:18:28.115>basic <00:18:28.290>use <00:18:28.465>case,

383
00:18:30.585 --> 00:18:32.485
<00:18:30.585>but <00:18:30.865>we <00:18:31.145>soon <00:18:31.425>got <00:18:31.705>another <00:18:31.985>requirement.

384
00:18:34.265 --> 00:18:38.165
<00:18:34.265>Um, <00:18:34.392>the <00:18:34.519>requirement <00:18:34.646>was <00:18:34.774>to <00:18:34.901>add <00:18:35.028>a <00:18:35.155>new <00:18:35.283>field <00:18:35.410>to <00:18:35.537>each <00:18:35.665>item,

385
00:18:38.175 --> 00:18:39.525
<00:18:38.175>which <00:18:38.580>we <00:18:38.985>call <00:18:39.390>nickname.

386
00:18:39.525 --> 00:18:42.205
<00:18:39.525>Here. <00:18:39.665>The <00:18:39.805>idea <00:18:39.945>is <00:18:40.085>that <00:18:40.225>there <00:18:40.365>is <00:18:40.505>a <00:18:40.645>user <00:18:40.785>editable <00:18:40.925>field,

387
00:18:42.615 --> 00:18:45.845
<00:18:42.615>which <00:18:42.790>lets <00:18:42.965>the <00:18:43.140>user <00:18:43.315>select <00:18:43.490>some <00:18:43.665>kind <00:18:43.840>of <00:18:44.015>name

388
00:18:45.865 --> 00:18:47.605
<00:18:45.865>to <00:18:46.331>identify <00:18:46.798>the <00:18:47.265>item.

389
00:18:48.425 --> 00:18:50.525
<00:18:48.425>And <00:18:48.658>the <00:18:48.891>business <00:18:49.125>rule <00:18:49.358>here <00:18:49.591>is <00:18:49.825>that

390
00:18:51.625 --> 00:18:53.605
<00:18:51.625>no <00:18:51.858>two <00:18:52.091>items <00:18:52.325>in <00:18:52.558>the <00:18:52.791>same <00:18:53.025>inventory

391
00:18:53.905 --> 00:18:55.085
<00:18:53.905>may <00:18:54.170>have <00:18:54.436>the <00:18:54.701>same <00:18:54.967>nickname.

392
00:18:55.945 --> 00:18:57.325
<00:18:55.945>So <00:18:56.193>how <00:18:56.441>do <00:18:56.690>we <00:18:56.938>implement <00:18:57.187>this?

393
00:18:58.745 --> 00:19:00.685
<00:18:58.745>Our <00:18:59.094>first <00:18:59.445>approach, <00:18:59.794>um, <00:19:00.145>was

394
00:19:00.685 --> 00:19:04.085
<00:19:00.685>to <00:19:00.860>use <00:19:01.034>a <00:19:01.210>Kafka <00:19:01.385>streams <00:19:01.560>feature <00:19:01.735>called <00:19:01.909>state <00:19:02.085>stores

395
00:19:05.065 --> 00:19:08.605
<00:19:05.065>to <00:19:05.240>read <00:19:05.415>the <00:19:05.590>inventory <00:19:05.765>items <00:19:05.940>topic <00:19:06.115>into <00:19:06.290>a <00:19:06.465>local

396
00:19:09.265 --> 00:19:11.845
<00:19:09.265>key <00:19:09.498>value <00:19:09.731>store <00:19:09.965>in <00:19:10.198>the <00:19:10.431>command <00:19:10.665>processor.

397
00:19:12.305 --> 00:19:14.765
<00:19:12.305>So <00:19:12.505>that <00:19:12.705>allows <00:19:12.905>the <00:19:13.105>processor <00:19:13.305>to <00:19:13.505>basically <00:19:13.705>check

398
00:19:14.765 --> 00:19:17.045
<00:19:14.765>for <00:19:15.045>every <00:19:15.325>incoming <00:19:15.605>creation <00:19:15.885>update <00:19:16.165>command

399
00:19:17.785 --> 00:19:20.445
<00:19:17.785>if <00:19:17.960>the <00:19:18.135>nickname <00:19:18.310>is <00:19:18.485>already <00:19:18.660>taken <00:19:18.835>in <00:19:19.010>the <00:19:19.185>inventory,

400
00:19:20.825 --> 00:19:22.925
<00:19:20.825>and <00:19:21.105>if <00:19:21.385>yes, <00:19:21.665>reject <00:19:21.945>the <00:19:22.225>command.

401
00:19:24.305 --> 00:19:26.285
<00:19:24.305>So <00:19:24.655>this <00:19:25.005>appeared <00:19:25.355>to <00:19:25.705>work,

402
00:19:26.345 --> 00:19:27.445
<00:19:26.345>but <00:19:26.592>we <00:19:26.840>soon <00:19:27.087>found <00:19:27.335>out

403
00:19:27.445 --> 00:19:29.485
<00:19:27.445>that <00:19:27.645>there <00:19:27.845>is <00:19:28.044>a <00:19:28.245>serious <00:19:28.445>race <00:19:28.645>condition <00:19:28.845>here.

404
00:19:31.015 --> 00:19:33.445
<00:19:31.015>Let's <00:19:31.190>imagine <00:19:31.365>we <00:19:31.540>have <00:19:31.715>a <00:19:31.890>new <00:19:32.065>command <00:19:32.240>coming <00:19:32.415>in

405
00:19:33.625 --> 00:19:36.685
<00:19:33.625>to <00:19:33.765>the <00:19:33.905>pipeline <00:19:34.045>to <00:19:34.185>add <00:19:34.325>some <00:19:34.465>new <00:19:34.605>item <00:19:34.745>with <00:19:34.885>some <00:19:35.025>nickname.

406
00:19:37.225 --> 00:19:40.005
<00:19:37.225>So <00:19:37.458>that <00:19:37.691>goes <00:19:37.925>into <00:19:38.158>the <00:19:38.391>command <00:19:38.625>processor,

407
00:19:40.135 --> 00:19:43.165
<00:19:40.135>which <00:19:40.290>then <00:19:40.446>checks <00:19:40.601>whether <00:19:40.757>a <00:19:40.912>nickname <00:19:41.068>is <00:19:41.223>taken, <00:19:41.379>let's <00:19:41.535>assume

408
00:19:43.165 --> 00:19:44.445
<00:19:43.165>that <00:19:43.453>the <00:19:43.741>nickname <00:19:44.029>is <00:19:44.317>free.

409
00:19:45.345 --> 00:19:47.565
<00:19:45.345>So <00:19:45.625>then <00:19:45.905>an <00:19:46.185>event <00:19:46.465>gets <00:19:46.745>created

410
00:19:48.465 --> 00:19:50.445
<00:19:48.465>and <00:19:48.745>goes <00:19:49.024>further <00:19:49.304>down <00:19:49.584>the <00:19:49.865>pipeline.

411
00:19:51.985 --> 00:19:53.685
<00:19:51.985>Now <00:19:52.218>what <00:19:52.451>can <00:19:52.685>happen <00:19:52.918>though <00:19:53.151>is <00:19:53.385>that

412
00:19:54.255 --> 00:19:57.645
<00:19:54.255>while <00:19:54.395>the <00:19:54.535>event <00:19:54.675>is <00:19:54.815>being <00:19:54.955>processed, <00:19:55.095>a <00:19:55.235>new <00:19:55.375>command <00:19:55.515>comes <00:19:55.655>in,

413
00:19:58.265 --> 00:20:00.885
<00:19:58.265>and <00:19:58.498>let's <00:19:58.731>assume <00:19:58.965>this <00:19:59.198>command's <00:19:59.431>actually <00:19:59.665>conflicting

414
00:20:00.885 --> 00:20:02.405
<00:20:00.885>with <00:20:01.158>the <00:20:01.432>previous <00:20:01.705>command, <00:20:01.979>so <00:20:02.253>tries

415
00:20:02.405 --> 00:20:03.645
<00:20:02.405>to <00:20:02.684>use <00:20:02.963>the <00:20:03.242>same <00:20:03.521>nickname.

416
00:20:05.265 --> 00:20:08.765
<00:20:05.265>If <00:20:05.440>that <00:20:05.615>command <00:20:05.790>is <00:20:05.965>processed <00:20:06.140>by <00:20:06.315>the <00:20:06.490>command <00:20:06.665>processor

417
00:20:09.025 --> 00:20:13.165
<00:20:09.025>before <00:20:09.200>the <00:20:09.375>inventory <00:20:09.550>item <00:20:09.725>from <00:20:09.900>the <00:20:10.075>previous <00:20:10.250>command <00:20:10.425>reached

418
00:20:13.225 --> 00:20:16.365
<00:20:13.225>the <00:20:13.399>state <00:20:13.574>store, <00:20:13.750>then <00:20:13.925>the <00:20:14.100>command <00:20:14.274>processor <00:20:14.449>will <00:20:14.625>believe

419
00:20:16.875 --> 00:20:19.845
<00:20:16.875>that <00:20:17.108>the <00:20:17.341>nickname <00:20:17.575>is <00:20:17.808>not <00:20:18.041>taken <00:20:18.275>yet,

420
00:20:20.465 --> 00:20:22.845
<00:20:20.465>and <00:20:20.698>it <00:20:20.931>will <00:20:21.165>actually <00:20:21.398>create <00:20:21.631>another <00:20:21.865>event

421
00:20:22.875 --> 00:20:24.165
<00:20:22.875>with <00:20:23.262>a <00:20:23.649>conflicting <00:20:24.036>item.

422
00:20:24.305 --> 00:20:26.685
<00:20:24.305>So <00:20:24.585>we <00:20:24.865>broke <00:20:25.145>our <00:20:25.425>uniqueness <00:20:25.705>constraint.

423
00:20:27.745 --> 00:20:30.045
<00:20:27.745>The <00:20:28.024>fundamental <00:20:28.304>problem <00:20:28.584>here <00:20:28.864>is <00:20:29.145>that

424
00:20:30.735 --> 00:20:32.445
<00:20:30.735>Kafka <00:20:31.084>is <00:20:31.435>inherently <00:20:31.784>a <00:20:32.135>distributed

425
00:20:32.505 --> 00:20:34.325
<00:20:32.505>and <00:20:32.971>eventually <00:20:33.438>consistent <00:20:33.905>system.

426
00:20:35.305 --> 00:20:37.125
<00:20:35.305>So <00:20:35.771>we <00:20:36.238>cannot <00:20:36.705>guarantee

427
00:20:37.155 --> 00:20:40.525
<00:20:37.155>that <00:20:37.355>the <00:20:37.555>previous <00:20:37.754>command <00:20:37.955>has <00:20:38.155>been <00:20:38.355>fully <00:20:38.555>processed

428
00:20:40.525 --> 00:20:42.285
<00:20:40.525>before <00:20:40.805>the <00:20:41.085>next <00:20:41.365>command <00:20:41.645>comes <00:20:41.925>in.

429
00:20:44.025 --> 00:20:45.125
<00:20:44.025>So <00:20:44.272>what <00:20:44.520>do <00:20:44.767>we <00:20:45.015>do?

430
00:20:45.955 --> 00:20:50.085
<00:20:45.955>Luckily, <00:20:46.110>uh, <00:20:46.266>Kafka <00:20:46.421>Streams <00:20:46.577>has <00:20:46.732>another <00:20:46.888>type <00:20:47.043>of <00:20:47.199>state <00:20:47.355>store,

431
00:20:50.085 --> 00:20:52.245
<00:20:50.085>which <00:20:50.285>is <00:20:50.485>not <00:20:50.685>popular <00:20:50.885>from <00:20:51.085>an <00:20:51.285>existing <00:20:51.485>topic,

432
00:20:52.945 --> 00:20:56.205
<00:20:52.945>but <00:20:53.100>instead, <00:20:53.256>you <00:20:53.411>can <00:20:53.567>actually <00:20:53.722>create <00:20:53.878>a <00:20:54.033>state <00:20:54.189>store <00:20:54.345>manually

433
00:20:56.785 --> 00:20:59.125
<00:20:56.785>and <00:20:57.065>update <00:20:57.345>it <00:20:57.625>manually <00:20:57.905>by <00:20:58.185>writing

434
00:20:59.225 --> 00:21:01.605
<00:20:59.225>to <00:20:59.458>the <00:20:59.691>on <00:20:59.925>the <00:21:00.158>state <00:21:00.391>store <00:21:00.625>synchronously.

435
00:21:02.465 --> 00:21:03.485
<00:21:02.465>So <00:21:02.771>that <00:21:03.077>allows <00:21:03.382>us

436
00:21:03.485 --> 00:21:06.085
<00:21:03.485>to <00:21:03.659>change <00:21:03.834>the <00:21:04.010>command <00:21:04.185>process <00:21:04.360>on <00:21:04.534>such <00:21:04.709>a <00:21:04.885>way

437
00:21:06.085 --> 00:21:10.365
<00:21:06.085>that <00:21:06.260>it <00:21:06.435>actually <00:21:06.610>on <00:21:06.785>every <00:21:06.960>command, <00:21:07.135>it <00:21:07.310>immediately <00:21:07.485>writes

438
00:21:10.365 --> 00:21:12.925
<00:21:10.365>to <00:21:10.520>the, <00:21:10.676>on <00:21:10.831>the <00:21:10.987>state <00:21:11.142>store <00:21:11.298>to <00:21:11.453>reserve <00:21:11.609>the <00:21:11.765>nickname

439
00:21:13.065 --> 00:21:14.685
<00:21:13.065>before <00:21:13.345>moving <00:21:13.625>to <00:21:13.905>the <00:21:14.185>next <00:21:14.465>command.

440
00:21:15.345 --> 00:21:16.445
<00:21:15.345>And <00:21:15.592>then <00:21:15.840>in <00:21:16.087>the <00:21:16.335>background,

441
00:21:16.655 --> 00:21:20.005
<00:21:16.655>Kafka <00:21:16.782>streams <00:21:16.909>will <00:21:17.036>back <00:21:17.164>up <00:21:17.291>the <00:21:17.418>state <00:21:17.545>store <00:21:17.673>to <00:21:17.800>a <00:21:17.927>Kafka <00:21:18.055>topic

442
00:21:20.105 --> 00:21:23.685
<00:21:20.105>for <00:21:20.385>availability <00:21:20.665>to <00:21:20.945>change <00:21:21.225>lock <00:21:21.505>topic.

443
00:21:24.625 --> 00:21:28.925
<00:21:24.625>So <00:21:24.765>that <00:21:24.905>way <00:21:25.045>we <00:21:25.185>remove <00:21:25.325>the <00:21:25.465>race <00:21:25.605>condition <00:21:25.745>from <00:21:25.885>our <00:21:26.025>pipeline.

444
00:21:28.945 --> 00:21:31.005
<00:21:28.945>So <00:21:29.145>instead <00:21:29.345>of <00:21:29.544>reading <00:21:29.745>the <00:21:29.945>inventory <00:21:30.145>items <00:21:30.345>topic,

445
00:21:31.065 --> 00:21:33.165
<00:21:31.065>we <00:21:31.298>have <00:21:31.531>this <00:21:31.765>manually <00:21:31.998>maintained <00:21:32.231>state <00:21:32.465>store,

446
00:21:33.535 --> 00:21:37.325
<00:21:33.535>which <00:21:33.710>does <00:21:33.885>increase <00:21:34.060>the <00:21:34.235>complexity <00:21:34.410>of <00:21:34.585>the <00:21:34.760>command <00:21:34.935>processor

447
00:21:37.325 --> 00:21:38.925
<00:21:37.325>because <00:21:37.500>now <00:21:37.675>it <00:21:37.850>has <00:21:38.025>to <00:21:38.200>manually <00:21:38.375>maintain <00:21:38.550>that <00:21:38.725>state,

448
00:21:39.465 --> 00:21:40.885
<00:21:39.465>but <00:21:39.720>the <00:21:39.976>race <00:21:40.231>condition <00:21:40.487>is <00:21:40.743>gone.

449
00:21:42.345 --> 00:21:46.565
<00:21:42.345>So <00:21:42.485>this <00:21:42.625>pipeline <00:21:42.765>worked <00:21:42.905>well <00:21:43.045>enough <00:21:43.185>when <00:21:43.325>we <00:21:43.465>just <00:21:43.605>had <00:21:43.745>a

450
00:21:46.565 --> 00:21:48.685
<00:21:46.565>single <00:21:46.798>instance <00:21:47.031>of <00:21:47.265>the <00:21:47.498>command <00:21:47.731>cell <00:21:47.965>running,

451
00:21:48.865 --> 00:21:50.885
<00:21:48.865>but <00:21:49.065>we <00:21:49.265>soon <00:21:49.465>found <00:21:49.665>out <00:21:49.865>that <00:21:50.065>we <00:21:50.265>need

452
00:21:50.885 --> 00:21:54.645
<00:21:50.885>to <00:21:51.040>do <00:21:51.196>something <00:21:51.351>in <00:21:51.507>addition <00:21:51.662>if <00:21:51.818>we <00:21:51.973>run <00:21:52.129>multiple <00:21:52.285>instances.

453
00:21:55.585 --> 00:22:00.325
<00:21:55.585>And <00:21:55.740>this <00:21:55.896>is <00:21:56.051>because <00:21:56.207>of <00:21:56.362>the <00:21:56.518>partitioning <00:21:56.673>concept <00:21:56.829>on <00:21:56.985>Kafka.

454
00:22:00.865 --> 00:22:03.445
<00:22:00.865>So <00:22:01.065>as <00:22:01.265>we <00:22:01.465>mentioned <00:22:01.665>in <00:22:01.865>the <00:22:02.065>introduction, <00:22:02.265>uh,

455
00:22:03.605 --> 00:22:06.685
<00:22:03.605>a <00:22:03.805>topic <00:22:04.005>may <00:22:04.205>consist <00:22:04.405>of <00:22:04.605>multiple <00:22:04.805>partitions, <00:22:05.005>each

456
00:22:06.685 --> 00:22:08.165
<00:22:06.685>of <00:22:06.907>which <00:22:07.129>has <00:22:07.351>part <00:22:07.572>of <00:22:07.795>the <00:22:08.017>data.

457
00:22:08.985 --> 00:22:12.365
<00:22:08.985>And <00:22:09.112>if <00:22:09.239>you <00:22:09.366>have <00:22:09.494>more <00:22:09.621>than <00:22:09.748>one <00:22:09.875>instance <00:22:10.003>of <00:22:10.130>a <00:22:10.257>Kafka <00:22:10.385>consumer

458
00:22:12.425 --> 00:22:15.005
<00:22:12.425>or <00:22:12.774>Kafka <00:22:13.125>streams <00:22:13.475>topology <00:22:13.825>running,

459
00:22:15.395 --> 00:22:17.365
<00:22:15.395>then <00:22:15.628>they <00:22:15.861>will <00:22:16.095>form <00:22:16.328>a <00:22:16.561>consumer <00:22:16.795>group.

460
00:22:17.545 --> 00:22:20.525
<00:22:17.545>So <00:22:17.895>each <00:22:18.245>partition <00:22:18.595>gets <00:22:18.945>assigned

461
00:22:20.525 --> 00:22:22.125
<00:22:20.525>to <00:22:20.805>exactly <00:22:21.085>one <00:22:21.365>of <00:22:21.645>the <00:22:21.925>instances.

462
00:22:22.425 --> 00:22:25.205
<00:22:22.425>So <00:22:22.565>that <00:22:22.705>means <00:22:22.845>every <00:22:22.985>instance <00:22:23.125>only <00:22:23.264>sees <00:22:23.405>part <00:22:23.544>of <00:22:23.685>the <00:22:23.825>data,

463
00:22:26.145 --> 00:22:28.085
<00:22:26.145>and <00:22:26.320>that <00:22:26.495>can <00:22:26.670>lead <00:22:26.845>to <00:22:27.020>problems <00:22:27.195>as <00:22:27.370>we <00:22:27.545>see.

464
00:22:29.105 --> 00:22:32.485
<00:22:29.105>So <00:22:29.245>let's <00:22:29.385>imagine <00:22:29.525>there's <00:22:29.665>a <00:22:29.805>new <00:22:29.945>command <00:22:30.085>from <00:22:30.225>the <00:22:30.365>rest, <00:22:30.505>API

465
00:22:32.545 --> 00:22:35.285
<00:22:32.545>to <00:22:32.778>update <00:22:33.011>some <00:22:33.245>items, <00:22:33.478>nickname <00:22:33.711>to <00:22:33.945>a

466
00:22:36.665 --> 00:22:39.845
<00:22:36.665>and <00:22:36.865>the <00:22:37.065>partition <00:22:37.264>where <00:22:37.465>this <00:22:37.665>command <00:22:37.865>goes, <00:22:38.065>um,

467
00:22:39.995 --> 00:22:42.925
<00:22:39.995>depends <00:22:40.169>on <00:22:40.344>the <00:22:40.520>key <00:22:40.695>of <00:22:40.870>the <00:22:41.044>record <00:22:41.219>by <00:22:41.395>default.

468
00:22:43.785 --> 00:22:47.245
<00:22:43.785>So <00:22:43.985>basically <00:22:44.185>now <00:22:44.385>the <00:22:44.585>first <00:22:44.785>petition <00:22:44.985>is <00:22:45.185>chosen

469
00:22:47.245 --> 00:22:49.365
<00:22:47.245>because <00:22:47.524>of <00:22:47.804>the <00:22:48.084>item <00:22:48.364>ID <00:22:48.645>1.1.

470
00:22:49.865 --> 00:22:52.325
<00:22:49.865>So <00:22:50.020>it <00:22:50.176>will <00:22:50.331>be <00:22:50.487>processed <00:22:50.642>by <00:22:50.798>the <00:22:50.953>first <00:22:51.109>command <00:22:51.265>processor,

471
00:22:52.325 --> 00:22:55.925
<00:22:52.325>which <00:22:52.480>will <00:22:52.636>then <00:22:52.791>update <00:22:52.947>its <00:22:53.102>state <00:22:53.258>store <00:22:53.413>of <00:22:53.569>reserve <00:22:53.725>nicknames.

472
00:22:57.825 --> 00:23:00.725
<00:22:57.825>But <00:22:57.952>what <00:22:58.079>can <00:22:58.206>happen <00:22:58.334>now <00:22:58.461>is <00:22:58.588>that <00:22:58.715>there <00:22:58.843>is <00:22:58.970>a <00:22:59.097>second <00:22:59.225>command

473
00:23:01.665 --> 00:23:04.005
<00:23:01.665>to <00:23:01.865>update <00:23:02.065>a <00:23:02.264>different <00:23:02.465>item <00:23:02.665>in <00:23:02.865>the <00:23:03.065>inventory

474
00:23:04.145 --> 00:23:05.205
<00:23:04.145>to <00:23:04.463>the <00:23:04.781>same <00:23:05.099>nickname.

475
00:23:06.105 --> 00:23:09.565
<00:23:06.105>Um, <00:23:06.232>but <00:23:06.359>because <00:23:06.486>the <00:23:06.614>item <00:23:06.741>ID <00:23:06.868>is <00:23:06.995>different, <00:23:07.123>what <00:23:07.250>can <00:23:07.377>happen <00:23:07.505>is

476
00:23:09.565 --> 00:23:10.725
<00:23:09.565>that <00:23:09.913>it <00:23:10.261>actually <00:23:10.609>goes

477
00:23:10.745 --> 00:23:13.205
<00:23:10.745>to <00:23:10.945>a <00:23:11.145>different <00:23:11.344>partition <00:23:11.544>on <00:23:11.745>the <00:23:11.945>commands <00:23:12.145>topic

478
00:23:13.665 --> 00:23:16.285
<00:23:13.665>and <00:23:13.865>what <00:23:14.065>we <00:23:14.264>process <00:23:14.465>by <00:23:14.665>a <00:23:14.865>different <00:23:15.065>processor,

479
00:23:16.855 --> 00:23:19.525
<00:23:16.855>which <00:23:17.135>hasn't <00:23:17.415>seen <00:23:17.695>the <00:23:17.975>previous <00:23:18.255>command.

480
00:23:20.505 --> 00:23:24.645
<00:23:20.505>So <00:23:20.632>it <00:23:20.759>will <00:23:20.886>not <00:23:21.014>see <00:23:21.141>the <00:23:21.268>entry <00:23:21.395>in <00:23:21.523>its <00:23:21.650>local <00:23:21.777>state <00:23:21.905>store.

481
00:23:25.105 --> 00:23:27.365
<00:23:25.105>We <00:23:25.260>have <00:23:25.416>to <00:23:25.571>remember <00:23:25.727>the <00:23:25.882>state <00:23:26.038>stores <00:23:26.193>are <00:23:26.349>on <00:23:26.505>disk.

482
00:23:27.425 --> 00:23:30.805
<00:23:27.425>So <00:23:27.600>each <00:23:27.774>instance <00:23:27.950>has <00:23:28.125>its <00:23:28.300>own <00:23:28.475>local <00:23:28.649>state <00:23:28.825>store

483
00:23:31.665 --> 00:23:33.885
<00:23:31.665>and <00:23:31.840>will <00:23:32.014>think <00:23:32.190>the <00:23:32.365>nickname <00:23:32.540>is <00:23:32.715>not <00:23:32.889>taken <00:23:33.065>yet

484
00:23:34.185 --> 00:23:35.925
<00:23:34.185>and <00:23:34.418>assign <00:23:34.651>it <00:23:34.885>to <00:23:35.118>that <00:23:35.351>other <00:23:35.585>item.

485
00:23:36.585 --> 00:23:40.045
<00:23:36.585>So <00:23:36.760>now <00:23:36.935>we <00:23:37.110>not <00:23:37.285>only <00:23:37.460>broke <00:23:37.635>the <00:23:37.810>uniqueness <00:23:37.985>constraint,

486
00:23:40.585 --> 00:23:43.965
<00:23:40.585>we <00:23:40.785>even <00:23:40.985>created <00:23:41.185>inconsistent <00:23:41.385>state <00:23:41.585>between <00:23:41.785>the <00:23:41.985>two

487
00:23:44.995 --> 00:23:46.325
<00:23:44.995>command <00:23:45.593>processor <00:23:46.192>instances.

488
00:23:48.145 --> 00:23:51.605
<00:23:48.145>And <00:23:48.320>the <00:23:48.495>underlying <00:23:48.670>problem <00:23:48.845>here <00:23:49.020>is <00:23:49.195>that <00:23:49.370>the <00:23:49.545>key

489
00:23:51.705 --> 00:23:56.125
<00:23:51.705>by <00:23:51.845>which <00:23:51.985>we <00:23:52.125>partition <00:23:52.264>the <00:23:52.405>topic <00:23:52.544>is <00:23:52.685>different <00:23:52.824>from <00:23:52.965>the <00:23:53.105>key

490
00:23:56.145 --> 00:23:58.005
<00:23:56.145>by <00:23:56.378>which <00:23:56.611>the <00:23:56.845>state <00:23:57.078>stores <00:23:57.311>are <00:23:57.545>partitioned.

491
00:23:58.105 --> 00:24:00.125
<00:23:58.105>So <00:23:58.455>we <00:23:58.805>cannot <00:23:59.155>guarantee <00:23:59.505>that

492
00:24:00.725 --> 00:24:03.165
<00:24:00.725>a <00:24:00.925>command <00:24:01.125>ends <00:24:01.324>up <00:24:01.524>in <00:24:01.725>the <00:24:01.925>command <00:24:02.125>processor,

493
00:24:03.165 --> 00:24:04.485
<00:24:03.165>which <00:24:03.462>has <00:24:03.759>the <00:24:04.055>relevant <00:24:04.352>data

494
00:24:04.545 --> 00:24:06.965
<00:24:04.545>to <00:24:04.778>make <00:24:05.011>the <00:24:05.245>decision <00:24:05.478>about <00:24:05.711>the <00:24:05.945>nickname.

495
00:24:08.185 --> 00:24:10.445
<00:24:08.185>So <00:24:08.340>what <00:24:08.496>we <00:24:08.651>need <00:24:08.807>to <00:24:08.962>do <00:24:09.118>is <00:24:09.273>actually <00:24:09.429>to <00:24:09.585>add

496
00:24:11.165 --> 00:24:12.605
<00:24:11.165>a <00:24:11.813>repartitioning <00:24:12.461>step

497
00:24:13.495 --> 00:24:18.125
<00:24:13.495>where <00:24:13.695>we <00:24:13.895>insert <00:24:14.094>another <00:24:14.294>Kafka <00:24:14.495>streams <00:24:14.695>processing <00:24:14.895>step

498
00:24:18.155 --> 00:24:20.325
<00:24:18.155>that <00:24:18.435>reads <00:24:18.715>from <00:24:18.995>the <00:24:19.274>commands <00:24:19.555>topic

499
00:24:21.065 --> 00:24:24.285
<00:24:21.065>and <00:24:21.298>then <00:24:21.531>chooses <00:24:21.765>a <00:24:21.998>new <00:24:22.231>key, <00:24:22.465>uh,

500
00:24:24.345 --> 00:24:26.325
<00:24:24.345>and <00:24:24.578>writes <00:24:24.811>that <00:24:25.045>to <00:24:25.278>an <00:24:25.511>intermediate <00:24:25.745>topic.

501
00:24:26.865 --> 00:24:28.965
<00:24:26.865>And <00:24:27.020>we <00:24:27.176>make <00:24:27.331>sure <00:24:27.487>that <00:24:27.642>this <00:24:27.798>key <00:24:27.953>matches <00:24:28.109>the <00:24:28.265>one

502
00:24:28.965 --> 00:24:30.045
<00:24:28.965>we <00:24:29.159>use <00:24:29.353>in <00:24:29.548>the <00:24:29.742>state <00:24:29.937>store.

503
00:24:30.105 --> 00:24:32.965
<00:24:30.105>So <00:24:30.280>in <00:24:30.455>this <00:24:30.630>case, <00:24:30.805>we <00:24:30.980>choose <00:24:31.155>the <00:24:31.330>inventory <00:24:31.505>id,

504
00:24:32.985 --> 00:24:34.925
<00:24:32.985>so <00:24:33.218>the <00:24:33.451>inventory <00:24:33.685>that <00:24:33.918>contains <00:24:34.151>the <00:24:34.385>item

505
00:24:35.355 --> 00:24:36.645
<00:24:35.355>because <00:24:35.587>we <00:24:35.819>know <00:24:36.051>that <00:24:36.283>we <00:24:36.516>have

506
00:24:36.645 --> 00:24:38.565
<00:24:36.645>to <00:24:36.878>enforce <00:24:37.111>the <00:24:37.345>unique <00:24:37.578>new <00:24:37.811>constraints <00:24:38.045>only

507
00:24:39.105 --> 00:24:40.405
<00:24:39.105>within <00:24:39.690>an <00:24:40.275>inventory.

508
00:24:41.145 --> 00:24:43.805
<00:24:41.145>So <00:24:41.345>with <00:24:41.545>this <00:24:41.745>change, <00:24:41.945>we <00:24:42.145>can <00:24:42.345>guarantee <00:24:42.545>that

509
00:24:44.315 --> 00:24:47.565
<00:24:44.315>each <00:24:44.470>command <00:24:44.626>adds <00:24:44.781>up, <00:24:44.937>ends <00:24:45.092>up <00:24:45.248>in <00:24:45.403>the <00:24:45.559>command <00:24:45.715>processor,

510
00:24:47.565 --> 00:24:50.165
<00:24:47.565>which <00:24:47.740>has <00:24:47.915>the <00:24:48.090>relevant <00:24:48.265>state <00:24:48.440>in <00:24:48.615>a <00:24:48.790>state <00:24:48.965>store.

511
00:24:52.185 --> 00:24:54.565
<00:24:52.185>So <00:24:52.325>this <00:24:52.465>is <00:24:52.605>the <00:24:52.745>pipeline <00:24:52.885>that <00:24:53.024>we <00:24:53.165>then <00:24:53.304>ended <00:24:53.445>up <00:24:53.585>with.

512
00:24:54.625 --> 00:24:58.085
<00:24:54.625>So <00:24:54.780>we <00:24:54.936>have <00:24:55.091>the <00:24:55.247>repartition <00:24:55.402>by <00:24:55.558>inventory <00:24:55.713>id, <00:24:55.869>then <00:24:56.025>the

513
00:24:58.725 --> 00:25:00.485
<00:24:58.725>manually <00:24:59.191>controlled <00:24:59.658>state <00:25:00.125>store,

514
00:25:00.585 --> 00:25:02.965
<00:25:00.585>and <00:25:00.760>then <00:25:00.935>we <00:25:01.110>actually <00:25:01.285>even <00:25:01.460>have <00:25:01.635>to <00:25:01.810>repartition <00:25:01.985>back

515
00:25:03.185 --> 00:25:04.885
<00:25:03.185>by <00:25:03.534>item <00:25:03.885>ID <00:25:04.235>because <00:25:04.585>that's

516
00:25:04.885 --> 00:25:08.925
<00:25:04.885>what <00:25:05.012>we <00:25:05.139>want <00:25:05.266>the <00:25:05.394>inventory <00:25:05.521>item <00:25:05.648>events <00:25:05.775>topic <00:25:05.903>to <00:25:06.030>be <00:25:06.157>key <00:25:06.285>by.

517
00:25:10.025 --> 00:25:14.485
<00:25:10.025>So <00:25:10.200>what <00:25:10.375>previously <00:25:10.550>looked <00:25:10.725>like <00:25:10.900>a <00:25:11.075>pretty <00:25:11.250>simple, <00:25:11.425>um,

518
00:25:14.875 --> 00:25:17.965
<00:25:14.875>requirement <00:25:15.075>actually <00:25:15.275>turned <00:25:15.475>out <00:25:15.675>to <00:25:15.875>increase <00:25:16.075>a <00:25:16.275>lot

519
00:25:17.965 --> 00:25:21.685
<00:25:17.965>of <00:25:18.165>complexity <00:25:18.365>if <00:25:18.564>done <00:25:18.764>with <00:25:18.965>a <00:25:19.165>pure <00:25:19.365>Kafka

520
00:25:21.685 --> 00:25:22.765
<00:25:21.685>and <00:25:22.009>Kafka <00:25:22.333>streams <00:25:22.657>approach,

521
00:25:23.215 --> 00:25:25.485
<00:25:23.215>which <00:25:23.370>would've <00:25:23.526>been <00:25:23.681>much <00:25:23.837>easier <00:25:23.992>if <00:25:24.148>we <00:25:24.303>had, <00:25:24.459>for <00:25:24.615>instance,

522
00:25:25.565 --> 00:25:27.565
<00:25:25.565>a <00:25:25.915>traditional <00:25:26.265>traditional <00:25:26.615>relation <00:25:26.965>database

523
00:25:27.935 --> 00:25:29.525
<00:25:27.935>where <00:25:28.168>we <00:25:28.401>could <00:25:28.635>have <00:25:28.868>just <00:25:29.101>created <00:25:29.335>a

524
00:25:29.525 --> 00:25:31.045
<00:25:29.525>uniqueness <00:25:29.753>concept <00:25:29.981>and <00:25:30.209>be <00:25:30.437>done <00:25:30.665>with <00:25:30.893>it.

525
00:25:33.945 --> 00:25:38.245
<00:25:33.945>All <00:25:34.100>right, <00:25:34.256>so, <00:25:34.411>um, <00:25:34.567>now <00:25:34.722>that <00:25:34.878>you've <00:25:35.033>seen <00:25:35.189>the, <00:25:35.345>uh,

526
00:25:38.765 --> 00:25:41.365
<00:25:38.765>difficulties <00:25:38.965>that <00:25:39.165>are <00:25:39.365>involved <00:25:39.565>in <00:25:39.765>this <00:25:39.965>inventory <00:25:40.165>pipeline,

527
00:25:41.815 --> 00:25:45.125
<00:25:41.815>let's <00:25:41.931>have <00:25:42.048>a <00:25:42.165>look <00:25:42.281>at <00:25:42.398>how <00:25:42.515>we <00:25:42.631>can <00:25:42.748>actually, <00:25:42.865>uh, <00:25:42.981>use <00:25:43.098>that <00:25:43.215>for,

528
00:25:45.225 --> 00:25:46.325
<00:25:45.225>for <00:25:45.720>something <00:25:46.215>interesting.

529
00:25:47.185 --> 00:25:50.125
<00:25:47.185>Um, <00:25:47.418>and <00:25:47.651>that's <00:25:47.885>the <00:25:48.118>automatic <00:25:48.351>warranty <00:25:48.585>extension.

530
00:25:50.945 --> 00:25:55.285
<00:25:50.945>So <00:25:51.100>in <00:25:51.256>order <00:25:51.411>to <00:25:51.567>register <00:25:51.722>warranty <00:25:51.878>extensions <00:25:52.033>for <00:25:52.189>the <00:25:52.345>products

531
00:25:55.285 --> 00:25:57.565
<00:25:55.285>that, <00:25:55.635>uh, <00:25:55.985>users <00:25:56.335>buy, <00:25:56.685>um,

532
00:25:57.755 --> 00:26:02.045
<00:25:57.755>what <00:25:57.910>we <00:25:58.066>need <00:25:58.221>is <00:25:58.377>information <00:25:58.532>about <00:25:58.688>their <00:25:58.843>inventory, <00:25:58.999>the <00:25:59.155>items

533
00:26:02.045 --> 00:26:03.045
<00:26:02.045>that <00:26:02.495>they've <00:26:02.945>registered,

534
00:26:03.825 --> 00:26:08.325
<00:26:03.825>and <00:26:04.000>we <00:26:04.175>also <00:26:04.350>need <00:26:04.525>information <00:26:04.700>about <00:26:04.875>the <00:26:05.050>products <00:26:05.225>themselves.

535
00:26:08.785 --> 00:26:11.165
<00:26:08.785>So <00:26:08.925>this <00:26:09.065>is <00:26:09.205>from <00:26:09.345>the <00:26:09.485>products <00:26:09.625>catalog <00:26:09.765>that <00:26:09.905>you <00:26:10.045>saw <00:26:10.185>earlier.

536
00:26:11.945 --> 00:26:15.325
<00:26:11.945>Uh, <00:26:12.100>this <00:26:12.256>is <00:26:12.411>needed <00:26:12.567>to <00:26:12.722>make, <00:26:12.878>for <00:26:13.033>example, <00:26:13.189>the <00:26:13.345>correct

537
00:26:15.845 --> 00:26:18.205
<00:26:15.845>warranty <00:26:16.311>extension <00:26:16.778>decisions, <00:26:17.245>uh,

538
00:26:18.205 --> 00:26:20.685
<00:26:18.205>which <00:26:18.405>are <00:26:18.605>often <00:26:18.804>based <00:26:19.004>on <00:26:19.205>on <00:26:19.405>product <00:26:19.605>model.

539
00:26:21.905 --> 00:26:25.325
<00:26:21.905>So <00:26:22.080>then <00:26:22.254>we <00:26:22.430>combine <00:26:22.605>these <00:26:22.780>two <00:26:22.955>data <00:26:23.129>streams <00:26:23.305>in

540
00:26:25.405 --> 00:26:26.565
<00:26:25.405>a <00:26:25.927>warranty <00:26:26.449>processor.

541
00:26:27.025 --> 00:26:28.805
<00:26:27.025>And <00:26:27.375>this <00:26:27.725>one, <00:26:28.075>uh, <00:26:28.425>listens

542
00:26:28.805 --> 00:26:31.965
<00:26:28.805>to <00:26:29.038>the <00:26:29.271>incoming <00:26:29.505>items <00:26:29.738>from <00:26:29.971>the <00:26:30.205>inventory

543
00:26:32.585 --> 00:26:35.325
<00:26:32.585>and <00:26:32.785>registers <00:26:32.985>a <00:26:33.185>warranty <00:26:33.385>extension <00:26:33.585>for <00:26:33.785>that <00:26:33.985>item.

544
00:26:35.705 --> 00:26:38.645
<00:26:35.705>If <00:26:35.938>it <00:26:36.171>passes <00:26:36.405>some <00:26:36.638>checks <00:26:36.871>around <00:26:37.105>eligibility.

545
00:26:39.625 --> 00:26:42.525
<00:26:39.625>Uh, <00:26:39.800>if <00:26:39.975>it <00:26:40.150>does, <00:26:40.325>the <00:26:40.500>warranty <00:26:40.675>extension <00:26:40.850>gets <00:26:41.025>written.

546
00:26:42.745 --> 00:26:44.285
<00:26:42.745>If <00:26:43.091>it <00:26:43.437>doesn't, <00:26:43.784>nothing <00:26:44.131>happens.

547
00:26:45.305 --> 00:26:49.925
<00:26:45.305>It <00:26:45.460>then <00:26:45.616>waits <00:26:45.771>or, <00:26:45.927>uh, <00:26:46.082>keeps <00:26:46.238>consuming <00:26:46.393>these <00:26:46.549>changed <00:26:46.705>items.

548
00:26:50.465 --> 00:26:54.125
<00:26:50.465>So <00:26:50.605>if, <00:26:50.745>for <00:26:50.885>example, <00:26:51.024>a <00:26:51.165>user <00:26:51.304>now <00:26:51.445>edits <00:26:51.584>their <00:26:51.725>item <00:26:51.865>adding,

549
00:26:54.225 --> 00:26:56.485
<00:26:54.225>for <00:26:54.425>example, <00:26:54.625>a <00:26:54.824>purchase <00:26:55.024>date <00:26:55.225>of <00:26:55.425>that <00:26:55.625>item,

550
00:26:56.485 --> 00:27:01.085
<00:26:56.485>which <00:26:56.625>is <00:26:56.764>needed <00:26:56.905>in <00:26:57.044>order <00:26:57.185>to <00:26:57.324>uh, <00:26:57.465>register <00:26:57.604>a <00:26:57.745>warranty, <00:26:57.885>um,

551
00:27:01.505 --> 00:27:04.645
<00:27:01.505>the <00:27:01.738>warranty <00:27:01.971>processor <00:27:02.205>automatically <00:27:02.438>sees <00:27:02.671>this <00:27:02.905>change.

552
00:27:05.265 --> 00:27:08.405
<00:27:05.265>And <00:27:05.420>if <00:27:05.576>it <00:27:05.731>passes <00:27:05.887>its <00:27:06.042>checks, <00:27:06.198>now <00:27:06.353>it <00:27:06.509>is <00:27:06.665>ready

553
00:27:08.405 --> 00:27:10.325
<00:27:08.405>to <00:27:08.754>write <00:27:09.105>the <00:27:09.455>warranty <00:27:09.805>extension.

554
00:27:10.785 --> 00:27:14.085
<00:27:10.785>What's <00:27:10.925>important <00:27:11.065>to <00:27:11.205>note <00:27:11.345>here <00:27:11.485>is <00:27:11.625>that <00:27:11.765>there's <00:27:11.905>no <00:27:12.045>such <00:27:12.185>thing

555
00:27:14.105 --> 00:27:15.405
<00:27:14.105>as <00:27:14.397>a <00:27:14.690>register <00:27:14.982>for <00:27:15.275>warranty.

556
00:27:15.445 --> 00:27:17.965
<00:27:15.445>API. <00:27:15.645>There <00:27:15.845>are <00:27:16.044>no <00:27:16.245>microservices <00:27:16.445>calling <00:27:16.645>each <00:27:16.845>other

557
00:27:18.065 --> 00:27:19.165
<00:27:18.065>and <00:27:18.395>querying <00:27:18.725>for <00:27:19.055>data.

558
00:27:20.035 --> 00:27:24.125
<00:27:20.035>This, <00:27:20.190>uh, <00:27:20.346>action <00:27:20.501>just <00:27:20.657>happens <00:27:20.812>as <00:27:20.968>soon <00:27:21.123>as <00:27:21.279>the <00:27:21.435>state

559
00:27:24.625 --> 00:27:26.845
<00:27:24.625>of <00:27:24.858>the <00:27:25.091>system <00:27:25.325>allows <00:27:25.558>it <00:27:25.791>to <00:27:26.025>happen.

560
00:27:28.145 --> 00:27:33.125
<00:27:28.145>So <00:27:28.300>this <00:27:28.456>allows <00:27:28.611>for <00:27:28.767>very <00:27:28.922>opportunistic <00:27:29.078>use <00:27:29.233>of <00:27:29.389>data <00:27:29.545>streams

561
00:27:33.305 --> 00:27:35.565
<00:27:33.305>by <00:27:33.538>different <00:27:33.771>teams <00:27:34.005>to <00:27:34.238>work <00:27:34.471>on <00:27:34.705>features.

562
00:27:35.565 --> 00:27:39.485
<00:27:35.565>In <00:27:35.705>parallel, <00:27:35.845>you <00:27:35.985>no <00:27:36.125>longer <00:27:36.265>need <00:27:36.405>to <00:27:36.545>do <00:27:36.685>so <00:27:36.825>much <00:27:36.965>collaboration

563
00:27:39.485 --> 00:27:42.525
<00:27:39.485>between <00:27:39.612>teams <00:27:39.739>to <00:27:39.866>figure <00:27:39.994>out <00:27:40.121>what <00:27:40.248>sort <00:27:40.375>of <00:27:40.503>messages <00:27:40.630>do <00:27:40.757>I <00:27:40.885>need

564
00:27:42.525 --> 00:27:45.405
<00:27:42.525>to <00:27:42.680>pass <00:27:42.836>to <00:27:42.991>whom <00:27:43.147>in <00:27:43.302>order <00:27:43.458>to <00:27:43.613>drive <00:27:43.769>certain <00:27:43.925>behavior.

565
00:27:45.945 --> 00:27:47.765
<00:27:45.945>And, <00:27:46.145>and <00:27:46.345>you <00:27:46.544>even <00:27:46.745>need <00:27:46.945>to <00:27:47.145>decide <00:27:47.345>when

566
00:27:47.825 --> 00:27:49.365
<00:27:47.825>to <00:27:48.171>send <00:27:48.518>those <00:27:48.864>uh, <00:27:49.211>messages.

567
00:27:50.225 --> 00:27:53.925
<00:27:50.225>In <00:27:50.399>this <00:27:50.574>case, <00:27:50.750>the <00:27:50.925>team <00:27:51.100>could <00:27:51.274>simply, <00:27:51.449>uh, <00:27:51.625>develop

568
00:27:54.465 --> 00:27:57.725
<00:27:54.465>and <00:27:54.639>then <00:27:54.814>deploy <00:27:54.990>this <00:27:55.165>warranty <00:27:55.340>processor <00:27:55.514>perhaps <00:27:55.689>one <00:27:55.865>year

569
00:27:57.815 --> 00:28:00.725
<00:27:57.815>after <00:27:58.015>these <00:27:58.215>products <00:27:58.415>and <00:27:58.615>items <00:27:58.815>have <00:27:59.015>been <00:27:59.215>collected.

570
00:28:01.185 --> 00:28:02.405
<00:28:01.185>And <00:28:01.404>it <00:28:01.624>would <00:28:01.843>then <00:28:02.063>be <00:28:02.283>able

571
00:28:02.405 --> 00:28:04.765
<00:28:02.405>to <00:28:02.638>read <00:28:02.871>both <00:28:03.105>topics <00:28:03.338>from <00:28:03.571>the <00:28:03.805>beginning

572
00:28:05.345 --> 00:28:06.765
<00:28:05.345>and <00:28:05.984>retroactively <00:28:06.623>even

573
00:28:07.485 --> 00:28:10.125
<00:28:07.485>register <00:28:07.764>these <00:28:08.044>warranty <00:28:08.324>extensions <00:28:08.604>if <00:28:08.885>desired.

574
00:28:11.225 --> 00:28:14.045
<00:28:11.225>And, <00:28:11.380>uh, <00:28:11.536>one <00:28:11.691>final <00:28:11.847>note <00:28:12.002>about <00:28:12.158>this <00:28:12.313>is <00:28:12.469>that <00:28:12.625>the,

575
00:28:14.105 --> 00:28:15.885
<00:28:14.105>the <00:28:14.385>data <00:28:14.665>stream <00:28:14.945>from <00:28:15.225>the <00:28:15.505>items

576
00:28:16.025 --> 00:28:17.845
<00:28:16.025>and <00:28:16.305>products <00:28:16.585>is <00:28:16.865>actually <00:28:17.145>the <00:28:17.425>data.

577
00:28:18.065 --> 00:28:19.245
<00:28:18.065>The <00:28:18.596>entities <00:28:19.127>themselves.

578
00:28:19.935 --> 00:28:24.645
<00:28:19.935>We're <00:28:20.075>not <00:28:20.215>sending <00:28:20.355>in <00:28:20.495>this <00:28:20.635>case, <00:28:20.774>uh, <00:28:20.915>events <00:28:21.054>like <00:28:21.195>item <00:28:21.335>added

579
00:28:25.665 --> 00:28:29.965
<00:28:25.665>or <00:28:25.820>item <00:28:25.976>updated, <00:28:26.131>we <00:28:26.287>are <00:28:26.442>actually <00:28:26.598>listening <00:28:26.753>to <00:28:26.909>the <00:28:27.065>entities,

580
00:28:30.065 --> 00:28:32.725
<00:28:30.065>the <00:28:30.220>whole <00:28:30.376>entities, <00:28:30.531>and <00:28:30.687>making <00:28:30.842>these <00:28:30.998>decisions <00:28:31.153>based <00:28:31.309>on <00:28:31.465>it.

581
00:28:32.945 --> 00:28:35.285
<00:28:32.945>So <00:28:33.178>you <00:28:33.411>can <00:28:33.645>even <00:28:33.878>drive <00:28:34.111>the <00:28:34.345>behavior

582
00:28:35.705 --> 00:28:38.525
<00:28:35.705>of <00:28:35.845>your <00:28:35.985>whole <00:28:36.125>system <00:28:36.264>based <00:28:36.405>on <00:28:36.544>these <00:28:36.685>sort <00:28:36.824>of <00:28:36.965>crud <00:28:37.105>events.

583
00:28:41.115 --> 00:28:43.965
<00:28:41.115>Okay, <00:28:41.270>so <00:28:41.426>now <00:28:41.581>you <00:28:41.737>saw <00:28:41.892>some <00:28:42.048>interesting <00:28:42.203>behavior <00:28:42.359>that, <00:28:42.515>uh,

584
00:28:43.965 --> 00:28:46.245
<00:28:43.965>this <00:28:44.165>enables, <00:28:44.365>but <00:28:44.564>also <00:28:44.764>the <00:28:44.965>pain <00:28:45.165>that's <00:28:45.365>involved.

585
00:28:46.875 --> 00:28:49.325
<00:28:46.875>What <00:28:47.075>did <00:28:47.275>we <00:28:47.475>learn? <00:28:47.675>So <00:28:47.875>first <00:28:48.075>of <00:28:48.275>all,

586
00:28:49.625 --> 00:28:52.245
<00:28:49.625>the <00:28:49.905>development <00:28:50.185>difficulty <00:28:50.465>is <00:28:50.745>quite <00:28:51.025>high.

587
00:28:52.635 --> 00:28:55.485
<00:28:52.635>It's <00:28:52.810>quite <00:28:52.985>difficult <00:28:53.160>to <00:28:53.335>onboard <00:28:53.510>new <00:28:53.685>developers <00:28:53.860>to <00:28:54.035>this.

588
00:28:56.145 --> 00:28:59.205
<00:28:56.145>You <00:28:56.300>really <00:28:56.456>need <00:28:56.611>to <00:28:56.767>plan <00:28:56.922>all <00:28:57.078>of <00:28:57.233>your <00:28:57.389>apologies <00:28:57.545>carefully.

589
00:28:59.705 --> 00:29:02.085
<00:28:59.705>Uh, <00:28:59.905>you, <00:29:00.105>you <00:29:00.304>need <00:29:00.504>to <00:29:00.705>think <00:29:00.905>about <00:29:01.105>keying

590
00:29:02.085 --> 00:29:03.765
<00:29:02.085>and <00:29:02.435>partitioning <00:29:02.785>in <00:29:03.135>that, <00:29:03.485>uh,

591
00:29:03.825 --> 00:29:06.445
<00:29:03.825>and <00:29:04.105>how <00:29:04.385>you're <00:29:04.665>handling <00:29:04.945>eventual <00:29:05.225>consistency.

592
00:29:06.745 --> 00:29:09.125
<00:29:06.745>Now, <00:29:06.919>of <00:29:07.094>course, <00:29:07.270>once <00:29:07.445>you've <00:29:07.620>done <00:29:07.794>that, <00:29:07.969>you <00:29:08.145>can

593
00:29:10.035 --> 00:29:12.565
<00:29:10.035>take <00:29:10.385>almost <00:29:10.735>one-to-one <00:29:11.085>your <00:29:11.435>designs

594
00:29:12.625 --> 00:29:15.925
<00:29:12.625>and <00:29:12.800>turn <00:29:12.975>them <00:29:13.150>into <00:29:13.325>Kafka <00:29:13.500>streams <00:29:13.675>code <00:29:13.850>relatively <00:29:14.025>easily.

595
00:29:17.585 --> 00:29:21.085
<00:29:17.585>But, <00:29:17.725>uh, <00:29:17.865>as <00:29:18.005>you <00:29:18.145>saw <00:29:18.285>then <00:29:18.425>with, <00:29:18.565>uh, <00:29:18.705>Dennis's <00:29:18.845>example, <00:29:18.985>uh,

596
00:29:21.095 --> 00:29:23.805
<00:29:21.095>developing <00:29:21.561>stateful <00:29:22.028>operations, <00:29:22.495>um,

597
00:29:24.115 --> 00:29:26.685
<00:29:24.115>enforcing <00:29:24.348>multiple <00:29:24.581>constraints <00:29:24.815>can <00:29:25.048>be <00:29:25.281>very <00:29:25.515>hard,

598
00:29:28.335 --> 00:29:32.245
<00:29:28.335>especially <00:29:28.510>if <00:29:28.685>we <00:29:28.860>are <00:29:29.035>talking <00:29:29.210>about <00:29:29.385>constraints <00:29:29.560>across, <00:29:29.735>uh,

599
00:29:32.685 --> 00:29:34.445
<00:29:32.685>multiple <00:29:33.034>different <00:29:33.385>types <00:29:33.735>of <00:29:34.085>entities.

600
00:29:35.985 --> 00:29:37.925
<00:29:35.985>Uh, <00:29:36.334>configuration <00:29:36.685>is <00:29:37.034>very <00:29:37.385>important.

601
00:29:38.425 --> 00:29:40.885
<00:29:38.425>Uh, <00:29:38.580>you <00:29:38.736>should <00:29:38.891>learn <00:29:39.047>all <00:29:39.202>the <00:29:39.358>config <00:29:39.513>values <00:29:39.669>very <00:29:39.825>well,

602
00:29:41.425 --> 00:29:43.045
<00:29:41.425>how <00:29:41.891>they <00:29:42.358>interact, <00:29:42.825>uh,

603
00:29:43.185 --> 00:29:46.325
<00:29:43.185>and <00:29:43.385>know <00:29:43.585>that <00:29:43.784>by <00:29:43.985>default, <00:29:44.185>uh, <00:29:44.385>these <00:29:44.585>Kafka

604
00:29:46.325 --> 00:29:47.885
<00:29:46.325>and <00:29:46.605>Kafka <00:29:46.885>streams <00:29:47.165>is <00:29:47.445>not <00:29:47.725>configured

605
00:29:47.885 --> 00:29:49.125
<00:29:47.885>for <00:29:48.071>the <00:29:48.257>use <00:29:48.443>cases <00:29:48.629>that <00:29:48.815>we've <00:29:49.001>shown.

606
00:29:49.815 --> 00:29:52.245
<00:29:49.815>Kafka <00:29:50.095>streams <00:29:50.375>is <00:29:50.655>actually <00:29:50.935>optimized <00:29:51.215>more

607
00:29:52.245 --> 00:29:53.885
<00:29:52.245>for <00:29:52.594>throughput <00:29:52.945>and <00:29:53.294>not <00:29:53.645>latency.

608
00:29:54.425 --> 00:29:56.645
<00:29:54.425>And <00:29:54.658>for <00:29:54.891>example, <00:29:55.125>uh, <00:29:55.358>when <00:29:55.591>it <00:29:55.825>comes

609
00:29:56.645 --> 00:29:57.925
<00:29:56.645>to <00:29:57.221>transactional <00:29:57.797>behavior,

610
00:29:58.425 --> 00:30:00.725
<00:29:58.425>we <00:29:58.625>shot <00:29:58.825>ourselves <00:29:59.024>in <00:29:59.225>the <00:29:59.425>foot <00:29:59.625>with <00:29:59.825>configuration.

611
00:30:01.265 --> 00:30:03.405
<00:30:01.265>So <00:30:01.545>we <00:30:01.825>had <00:30:02.105>a <00:30:02.385>transactional <00:30:02.665>producer,

612
00:30:03.655 --> 00:30:06.565
<00:30:03.655>which <00:30:03.888>is <00:30:04.121>writing <00:30:04.355>data <00:30:04.588>into <00:30:04.821>Kafka <00:30:05.055>topics,

613
00:30:07.185 --> 00:30:08.445
<00:30:07.185>uh, <00:30:07.752>with <00:30:08.319>transactions.

614
00:30:08.585 --> 00:30:11.805
<00:30:08.585>So <00:30:08.712>first <00:30:08.839>it <00:30:08.966>writes <00:30:09.094>the <00:30:09.221>data <00:30:09.348>and <00:30:09.475>then <00:30:09.603>it <00:30:09.730>writes <00:30:09.857>a <00:30:09.985>commit.

615
00:30:12.705 --> 00:30:16.845
<00:30:12.705>The <00:30:12.905>problem <00:30:13.105>here <00:30:13.304>is <00:30:13.504>that <00:30:13.705>by <00:30:13.905>default, <00:30:14.105>consumers,

616
00:30:17.585 --> 00:30:19.525
<00:30:17.585>uh, <00:30:18.051>actually <00:30:18.518>are <00:30:18.985>configured

617
00:30:19.745 --> 00:30:22.925
<00:30:19.745>to <00:30:19.978>read <00:30:20.211>uncommitted <00:30:20.445>data <00:30:20.678>in <00:30:20.911>those <00:30:21.145>topics.

618
00:30:23.945 --> 00:30:25.005
<00:30:23.945>So <00:30:24.135>you <00:30:24.326>have <00:30:24.517>to <00:30:24.708>be <00:30:24.899>careful

619
00:30:25.185 --> 00:30:26.885
<00:30:25.185>to <00:30:25.418>make <00:30:25.651>sure <00:30:25.885>you've <00:30:26.118>configured <00:30:26.351>your <00:30:26.585>consumer

620
00:30:26.985 --> 00:30:29.245
<00:30:26.985>to <00:30:27.264>also <00:30:27.544>only <00:30:27.824>read <00:30:28.104>committed <00:30:28.385>data.

621
00:30:29.825 --> 00:30:32.365
<00:30:29.825>Uh, <00:30:29.952>we, <00:30:30.079>we <00:30:30.206>ran <00:30:30.334>into <00:30:30.461>like <00:30:30.588>we <00:30:30.715>didn't <00:30:30.843>even <00:30:30.970>know <00:30:31.097>this <00:30:31.225>was

622
00:30:32.365 --> 00:30:36.485
<00:30:32.365>happening, <00:30:32.565>but <00:30:32.765>consumers, <00:30:32.965>some <00:30:33.165>consumers <00:30:33.365>were <00:30:33.565>writing, <00:30:33.765>uh,

623
00:30:36.635 --> 00:30:39.125
<00:30:36.635>data <00:30:36.915>downstream <00:30:37.195>when <00:30:37.475>it <00:30:37.754>wasn't <00:30:38.035>committed.

624
00:30:39.985 --> 00:30:43.605
<00:30:39.985>And <00:30:40.140>we <00:30:40.296>noticed <00:30:40.451>this <00:30:40.607>once <00:30:40.762>a <00:30:40.918>transactional <00:30:41.073>producer <00:30:41.229>crashed <00:30:41.385>in

625
00:30:43.605 --> 00:30:44.685
<00:30:43.605>the <00:30:43.848>middle <00:30:44.091>of <00:30:44.334>a <00:30:44.577>commit.

626
00:30:45.225 --> 00:30:49.165
<00:30:45.225>So <00:30:45.352>what <00:30:45.479>we <00:30:45.606>ended <00:30:45.734>up <00:30:45.861>with <00:30:45.988>was <00:30:46.115>some <00:30:46.243>data <00:30:46.370>in <00:30:46.497>downstream <00:30:46.625>topics

627
00:30:49.195 --> 00:30:51.605
<00:30:49.195>that <00:30:49.395>we <00:30:49.595>couldn't <00:30:49.794>actually <00:30:49.995>find <00:30:50.195>in <00:30:50.395>the <00:30:50.595>upstream

628
00:30:51.605 --> 00:30:55.725
<00:30:51.605>source <00:30:51.955>ops <00:30:52.305>is <00:30:52.655>quite <00:30:53.005>hard.

629
00:30:56.065 --> 00:30:58.125
<00:30:56.065>Um, <00:30:56.265>so <00:30:56.465>the <00:30:56.665>tooling <00:30:56.865>is <00:30:57.065>very <00:30:57.265>bare <00:30:57.465>bones

630
00:30:58.185 --> 00:31:00.365
<00:30:58.185>and <00:30:58.360>you're <00:30:58.534>very <00:30:58.710>likely <00:30:58.885>to <00:30:59.060>have <00:30:59.235>to <00:30:59.409>write <00:30:59.585>some

631
00:31:00.365 --> 00:31:01.605
<00:31:00.365>of <00:31:00.644>your <00:31:00.923>own <00:31:01.202>in <00:31:01.481>order

632
00:31:01.665 --> 00:31:03.845
<00:31:01.665>to <00:31:01.898>make <00:31:02.131>development <00:31:02.365>easier <00:31:02.598>and <00:31:02.831>ops <00:31:03.065>easier.

633
00:31:04.785 --> 00:31:06.445
<00:31:04.785>Uh, <00:31:05.135>for <00:31:05.485>example, <00:31:05.835>inspecting <00:31:06.185>data

634
00:31:06.465 --> 00:31:08.725
<00:31:06.465>and <00:31:06.698>topics <00:31:06.931>is, <00:31:07.165>is <00:31:07.398>very <00:31:07.631>difficult, <00:31:07.865>uh,

635
00:31:08.725 --> 00:31:09.845
<00:31:08.725>especially <00:31:08.926>if <00:31:09.128>you're <00:31:09.329>trying <00:31:09.531>to <00:31:09.733>look

636
00:31:09.865 --> 00:31:11.285
<00:31:09.865>for <00:31:10.120>certain <00:31:10.376>values <00:31:10.631>here <00:31:10.887>and <00:31:11.143>there.

637
00:31:11.945 --> 00:31:15.005
<00:31:11.945>Uh, <00:31:12.100>fixing <00:31:12.256>corrupted <00:31:12.411>data <00:31:12.567>in <00:31:12.722>a <00:31:12.878>source <00:31:13.033>of <00:31:13.189>truth <00:31:13.345>topic,

638
00:31:15.105 --> 00:31:17.005
<00:31:15.105>it <00:31:15.385>can <00:31:15.665>be <00:31:15.945>very <00:31:16.225>painful, <00:31:16.505>uh,

639
00:31:17.005 --> 00:31:19.645
<00:31:17.005>because <00:31:17.238>you <00:31:17.471>might <00:31:17.705>even <00:31:17.938>need <00:31:18.171>to <00:31:18.405>migrate

640
00:31:19.915 --> 00:31:22.645
<00:31:19.915>that <00:31:20.115>whole <00:31:20.315>topic <00:31:20.514>in <00:31:20.715>order <00:31:20.915>to <00:31:21.115>get <00:31:21.315>back

641
00:31:22.645 --> 00:31:24.005
<00:31:22.645>to <00:31:22.951>a <00:31:23.257>non <00:31:23.563>erupted <00:31:23.869>state.

642
00:31:25.305 --> 00:31:27.965
<00:31:25.305>You <00:31:25.460>have <00:31:25.616>potentially <00:31:25.771>long <00:31:25.927>startup <00:31:26.082>times <00:31:26.238>if <00:31:26.393>you <00:31:26.549>have <00:31:26.705>very

643
00:31:27.965 --> 00:31:30.845
<00:31:27.965>large <00:31:28.139>caches <00:31:28.314>that, <00:31:28.490>uh, <00:31:28.665>need <00:31:28.840>to <00:31:29.014>be <00:31:29.189>warmed <00:31:29.365>up.

644
00:31:30.915 --> 00:31:34.125
<00:31:30.915>When <00:31:31.090>you <00:31:31.264>spin <00:31:31.440>up <00:31:31.615>your <00:31:31.790>microservice <00:31:31.965>containing <00:31:32.139>the <00:31:32.315>Kafka

645
00:31:34.125 --> 00:31:38.165
<00:31:34.125>streams, <00:31:34.358>uh, <00:31:34.591>applications, <00:31:34.825>memory <00:31:35.058>usage <00:31:35.291>is <00:31:35.525>hard

646
00:31:38.165 --> 00:31:41.925
<00:31:38.165>to <00:31:38.365>configure <00:31:38.565>correctly <00:31:38.764>and <00:31:38.965>also <00:31:39.165>pick <00:31:39.365>Kafka <00:31:39.565>streams.

647
00:31:41.925 --> 00:31:45.605
<00:31:41.925>Applications, <00:31:42.158>um, <00:31:42.391>are <00:31:42.625>problematic <00:31:42.858>as <00:31:43.091>we <00:31:43.325>discovered.

648
00:31:46.305 --> 00:31:47.765
<00:31:46.305>So <00:31:46.743>here's <00:31:47.181>an <00:31:47.619>example.

649
00:31:47.945 --> 00:31:50.325
<00:31:47.945>Um, <00:31:48.085>we <00:31:48.225>have <00:31:48.365>a <00:31:48.504>microservice, <00:31:48.645>this <00:31:48.784>is <00:31:48.925>how <00:31:49.064>we <00:31:49.205>started <00:31:49.345>out.

650
00:31:50.585 --> 00:31:54.325
<00:31:50.585>We <00:31:50.785>usually <00:31:50.985>deployed <00:31:51.185>one <00:31:51.385>Kafka <00:31:51.585>streams <00:31:51.785>application <00:31:51.985>per

651
00:31:54.325 --> 00:31:56.925
<00:31:54.325>microservice, <00:31:54.525>and <00:31:54.725>a <00:31:54.925>Kafka <00:31:55.125>streams <00:31:55.325>application <00:31:55.525>is <00:31:55.725>here

652
00:31:56.945 --> 00:31:59.205
<00:31:56.945>to <00:31:57.178>noted <00:31:57.411>by <00:31:57.645>the <00:31:57.878>round <00:31:58.111>and <00:31:58.345>rectangle.

653
00:32:00.345 --> 00:32:04.725
<00:32:00.345>Um, <00:32:00.485>but <00:32:00.625>the <00:32:00.765>problem <00:32:00.905>that <00:32:01.045>happens <00:32:01.185>if <00:32:01.325>you <00:32:01.465>do <00:32:01.605>that <00:32:01.745>is

654
00:32:04.725 --> 00:32:07.605
<00:32:04.725>that <00:32:04.880>you <00:32:05.036>have <00:32:05.191>sub <00:32:05.347>apologies <00:32:05.502>in <00:32:05.658>there, <00:32:05.813>multiple <00:32:05.969>sub <00:32:06.125>apologies

655
00:32:07.605 --> 00:32:08.845
<00:32:07.605>that <00:32:07.977>have <00:32:08.349>different <00:32:08.721>needs.

656
00:32:09.705 --> 00:32:13.205
<00:32:09.705>So <00:32:09.845>if, <00:32:09.985>for <00:32:10.125>example, <00:32:10.264>you <00:32:10.405>have <00:32:10.544>one <00:32:10.685>sub <00:32:10.824>topology <00:32:10.965>that <00:32:11.105>needs

657
00:32:13.205 --> 00:32:17.445
<00:32:13.205>to <00:32:13.360>be <00:32:13.516>transactional <00:32:13.671>and <00:32:13.827>have <00:32:13.982>exactly <00:32:14.138>one <00:32:14.293>semantics, <00:32:14.449>you <00:32:14.605>need

658
00:32:17.445 --> 00:32:20.005
<00:32:17.445>to <00:32:17.678>configure <00:32:17.911>your <00:32:18.145>whole <00:32:18.378>Kafka <00:32:18.611>streams <00:32:18.845>application

659
00:32:20.105 --> 00:32:21.165
<00:32:20.105>to <00:32:20.581>be <00:32:21.059>transactional

660
00:32:21.395 --> 00:32:25.245
<00:32:21.395>because, <00:32:21.628>uh, <00:32:21.861>a <00:32:22.095>Kafka <00:32:22.328>Streams <00:32:22.561>application <00:32:22.795>shares

661
00:32:25.845 --> 00:32:27.325
<00:32:25.845>configuration <00:32:26.111>across <00:32:26.377>all <00:32:26.644>of <00:32:26.910>it, <00:32:27.177>all

662
00:32:27.325 --> 00:32:28.445
<00:32:27.325>of <00:32:27.577>its, <00:32:27.829>uh, <00:32:28.081>sub <00:32:28.332>topologies.

663
00:32:29.705 --> 00:32:32.365
<00:32:29.705>So <00:32:29.905>one <00:32:30.105>big <00:32:30.304>improvement <00:32:30.504>that <00:32:30.705>we <00:32:30.905>did <00:32:31.105>was

664
00:32:32.545 --> 00:32:35.245
<00:32:32.545>to <00:32:32.825>split <00:32:33.105>into <00:32:33.385>multiple <00:32:33.665>Kafka <00:32:33.945>streams

665
00:32:35.245 --> 00:32:36.765
<00:32:35.245>applications <00:32:35.929>per <00:32:36.613>microservice.

666
00:32:37.385 --> 00:32:39.765
<00:32:37.385>So <00:32:37.585>whenever <00:32:37.785>we <00:32:37.985>had <00:32:38.185>an <00:32:38.385>independent <00:32:38.585>sub <00:32:38.785>topology,

667
00:32:40.065 --> 00:32:43.325
<00:32:40.065>we <00:32:40.220>would <00:32:40.376>give <00:32:40.531>it <00:32:40.687>to <00:32:40.842>its <00:32:40.998>own <00:32:41.153>Kafka <00:32:41.309>Streams <00:32:41.465>application.

668
00:32:44.025 --> 00:32:47.725
<00:32:44.025>And <00:32:44.141>what <00:32:44.258>this <00:32:44.375>allowed <00:32:44.491>us <00:32:44.608>to <00:32:44.725>do <00:32:44.841>was <00:32:44.958>it <00:32:45.075>allowed <00:32:45.191>us <00:32:45.308>to, <00:32:45.425>um,

669
00:32:48.505 --> 00:32:51.765
<00:32:48.505>to <00:32:48.645>fine <00:32:48.785>tuning <00:32:48.925>based <00:32:49.065>on <00:32:49.205>the <00:32:49.345>needs <00:32:49.485>of <00:32:49.625>the <00:32:49.765>sub <00:32:49.905>topology.

670
00:32:51.865 --> 00:32:53.725
<00:32:51.865>So <00:32:52.065>if <00:32:52.265>it's <00:32:52.465>transactional, <00:32:52.665>we <00:32:52.865>can <00:32:53.065>configure <00:32:53.265>it.

671
00:32:53.825 --> 00:32:56.845
<00:32:53.825>So <00:32:54.000>if <00:32:54.175>it <00:32:54.350>only <00:32:54.525>needs <00:32:54.700>at <00:32:54.875>least <00:32:55.050>one <00:32:55.225>semantics,

672
00:32:57.105 --> 00:32:58.565
<00:32:57.105>we <00:32:57.367>can <00:32:57.630>configure <00:32:57.893>it <00:32:58.156>that <00:32:58.419>way.

673
00:32:58.915 --> 00:33:01.485
<00:32:58.915>This <00:32:59.148>way, <00:32:59.381>for <00:32:59.615>example, <00:32:59.848>this <00:33:00.081>transactional, <00:33:00.315>uh,

674
00:33:02.085 --> 00:33:04.245
<00:33:02.085>overhead <00:33:02.435>doesn't <00:33:02.785>propagate <00:33:03.135>everywhere <00:33:03.485>and,

675
00:33:04.245 --> 00:33:05.245
<00:33:04.245>and <00:33:04.469>slow <00:33:04.695>down <00:33:04.919>your <00:33:05.145>system,

676
00:33:05.505 --> 00:33:09.005
<00:33:05.505>you <00:33:05.660>can <00:33:05.816>also <00:33:05.971>then <00:33:06.127>more <00:33:06.282>carefully <00:33:06.438>fine <00:33:06.593>tune <00:33:06.749>your <00:33:06.905>latency

677
00:33:09.005 --> 00:33:11.165
<00:33:09.005>versus <00:33:09.355>throughput <00:33:09.705>compromises <00:33:10.055>and <00:33:10.405>needs.

678
00:33:11.755 --> 00:33:14.525
<00:33:11.755>This <00:33:11.910>also <00:33:12.066>actually <00:33:12.221>made <00:33:12.377>it, <00:33:12.532>uh, <00:33:12.688>much <00:33:12.843>easier <00:33:12.999>for <00:33:13.155>us

679
00:33:14.525 --> 00:33:19.045
<00:33:14.525>to <00:33:14.680>avoid <00:33:14.836>downtime <00:33:14.991>when <00:33:15.147>we <00:33:15.302>deployed <00:33:15.458>larger <00:33:15.613>changes <00:33:15.769>to, <00:33:15.925>uh,

680
00:33:19.185 --> 00:33:21.125
<00:33:19.185>sub <00:33:19.534>topologies <00:33:19.885>in <00:33:20.235>Kafka <00:33:20.585>streams.

681
00:33:23.195 --> 00:33:25.605
<00:33:23.195>Alright, <00:33:23.370>yeah, <00:33:23.544>before <00:33:23.720>we <00:33:23.895>wrap <00:33:24.070>up, <00:33:24.245>we <00:33:24.419>would <00:33:24.595>like

682
00:33:25.605 --> 00:33:28.245
<00:33:25.605>to <00:33:25.745>show <00:33:25.885>you <00:33:26.025>how <00:33:26.165>we <00:33:26.305>are <00:33:26.445>trying <00:33:26.585>to <00:33:26.725>evolve <00:33:26.865>the <00:33:27.005>system

683
00:33:28.305 --> 00:33:30.485
<00:33:28.305>to <00:33:28.538>tackle <00:33:28.771>some <00:33:29.005>of <00:33:29.238>the <00:33:29.471>complexity <00:33:29.705>issues

684
00:33:30.485 --> 00:33:34.085
<00:33:30.485>that <00:33:30.625>we <00:33:30.764>saw <00:33:30.905>in <00:33:31.044>the <00:33:31.185>inventory <00:33:31.324>management <00:33:31.465>example <00:33:31.604>where <00:33:31.745>we <00:33:31.885>saw

685
00:33:34.085 --> 00:33:36.485
<00:33:34.085>that <00:33:34.260>things <00:33:34.435>that <00:33:34.610>would <00:33:34.785>be <00:33:34.960>pretty <00:33:35.135>easy <00:33:35.310>to <00:33:35.485>do

686
00:33:36.485 --> 00:33:40.005
<00:33:36.485>with <00:33:36.718>a <00:33:36.951>traditional <00:33:37.185>relation <00:33:37.418>database <00:33:37.651>like <00:33:37.885>uniqueness

687
00:33:40.005 --> 00:33:43.045
<00:33:40.005>constraints <00:33:40.205>are <00:33:40.405>pretty <00:33:40.605>complicated <00:33:40.805>when <00:33:41.005>you <00:33:41.205>just <00:33:41.405>have

688
00:33:43.215 --> 00:33:44.405
<00:33:43.215>Kafka <00:33:43.572>and <00:33:43.928>Kafka <00:33:44.286>streams.

689
00:33:45.505 --> 00:33:49.845
<00:33:45.505>So <00:33:45.660>we <00:33:45.816>started <00:33:45.971>investigating, <00:33:46.127>uh, <00:33:46.282>how <00:33:46.438>we <00:33:46.593>could <00:33:46.749>maybe, <00:33:46.905>uh,

690
00:33:49.875 --> 00:33:53.845
<00:33:49.875>integrate <00:33:50.108>such <00:33:50.341>classic <00:33:50.575>databases <00:33:50.808>into <00:33:51.041>our <00:33:51.275>architecture

691
00:33:54.355 --> 00:33:57.525
<00:33:54.355>without <00:33:54.555>changing <00:33:54.755>anything <00:33:54.955>about <00:33:55.155>the <00:33:55.355>event <00:33:55.555>driven <00:33:55.755>approach

692
00:33:57.625 --> 00:33:59.645
<00:33:57.625>and <00:33:57.905>having <00:33:58.185>everything <00:33:58.465>published <00:33:58.745>in <00:33:59.025>Kafka.

693
00:34:00.625 --> 00:34:02.445
<00:34:00.625>So <00:34:00.858>what <00:34:01.091>we <00:34:01.325>plan <00:34:01.558>to <00:34:01.791>do <00:34:02.025>is

694
00:34:02.545 --> 00:34:06.005
<00:34:02.545>to <00:34:02.745>actually <00:34:02.945>have <00:34:03.145>processors <00:34:03.345>like <00:34:03.545>the <00:34:03.745>inventory <00:34:03.945>processor

695
00:34:06.635 --> 00:34:10.725
<00:34:06.635>that <00:34:06.810>internally <00:34:06.985>uses <00:34:07.160>a <00:34:07.335>database <00:34:07.510>to <00:34:07.685>implement <00:34:07.860>its <00:34:08.034>logic,

696
00:34:11.465 --> 00:34:14.685
<00:34:11.465>but <00:34:11.640>we <00:34:11.815>would <00:34:11.990>have, <00:34:12.165>um, <00:34:12.340>a <00:34:12.515>connector <00:34:12.690>that <00:34:12.865>listens

697
00:34:14.685 --> 00:34:17.725
<00:34:14.685>to <00:34:14.918>database <00:34:15.151>changes <00:34:15.384>and <00:34:15.618>streams <00:34:15.851>out <00:34:16.085>all

698
00:34:17.725 --> 00:34:20.285
<00:34:17.725>of <00:34:17.924>the <00:34:18.125>database <00:34:18.324>contents <00:34:18.525>again <00:34:18.725>to <00:34:18.924>Kafka <00:34:19.125>topics.

699
00:34:21.305 --> 00:34:23.885
<00:34:21.304>So <00:34:21.504>what <00:34:21.705>you <00:34:21.904>basically <00:34:22.105>still <00:34:22.304>see <00:34:22.504>from <00:34:22.705>the

700
00:34:23.885 --> 00:34:27.285
<00:34:23.885>outside <00:34:24.085>from <00:34:24.285>the <00:34:24.485>architectural <00:34:24.685>standpoint <00:34:24.885>is <00:34:25.085>a <00:34:25.285>processor

701
00:34:27.285 --> 00:34:29.605
<00:34:27.284>that <00:34:27.484>reads <00:34:27.685>Kafka <00:34:27.884>topics <00:34:28.085>and <00:34:28.284>writes <00:34:28.484>Kafka <00:34:28.685>topics.

702
00:34:30.425 --> 00:34:34.045
<00:34:30.425>But <00:34:30.625>the <00:34:30.825>internal <00:34:31.025>implementation <00:34:31.225>can <00:34:31.425>be <00:34:31.625>much <00:34:31.825>simpler

703
00:34:34.045 --> 00:34:35.765
<00:34:34.045>because <00:34:34.278>it <00:34:34.511>can <00:34:34.745>make <00:34:34.978>use <00:34:35.211>of <00:34:35.445>all

704
00:34:35.765 --> 00:34:38.245
<00:34:35.764>of <00:34:35.940>these <00:34:36.114>database <00:34:36.290>features <00:34:36.464>we <00:34:36.639>don't <00:34:36.815>have <00:34:36.989>with <00:34:37.165>Kafka.

705
00:34:39.265 --> 00:34:42.005
<00:34:39.264>So <00:34:39.464>yeah, <00:34:39.665>this <00:34:39.864>is <00:34:40.065>something <00:34:40.264>we're <00:34:40.464>currently <00:34:40.665>implementing

706
00:34:42.025 --> 00:34:44.325
<00:34:42.025>and <00:34:42.152>we're <00:34:42.279>looking <00:34:42.406>forward <00:34:42.534>to <00:34:42.661>how <00:34:42.788>it <00:34:42.915>will <00:34:43.043>go <00:34:43.170>and <00:34:43.297>help <00:34:43.425>us.

707
00:34:46.075 --> 00:34:48.405
<00:34:46.074>Alright, <00:34:46.355>that's <00:34:46.634>it <00:34:46.915>from <00:34:47.194>us. <00:34:47.475>Yeah,

708
00:34:48.405 --> 00:34:49.405
<00:34:48.405>Thanks. <00:34:48.855>Yeah, <00:34:49.305>thank

709
00:34:49.405 --> 00:34:51.325
<00:34:49.405>you <00:34:49.638>and <00:34:49.871>looking <00:34:50.105>forward <00:34:50.338>to <00:34:50.571>your <00:34:50.805>questions.

710
00:34:59.815 --> 00:35:02.805
<00:34:59.815>Hello. <00:35:00.095>Hello. <00:35:00.375>Thank <00:35:00.655>you. <00:35:00.935>Thank <00:35:01.215>you.

711
00:35:03.195 --> 00:35:04.445
<00:35:03.195>Very <00:35:03.476>nice, <00:35:03.757>very <00:35:04.038>nice <00:35:04.320>talk.

712
00:35:07.625 --> 00:35:09.765
<00:35:07.625>Uh, <00:35:07.975>the <00:35:08.325>top <00:35:08.675>question <00:35:09.025>is

713
00:35:10.865 --> 00:35:12.885
<00:35:10.865>can <00:35:11.145>you <00:35:11.424>provide <00:35:11.705>your <00:35:11.984>presentation <00:35:12.265>slides?

714
00:35:15.035 --> 00:35:17.335
<00:35:15.035>I'm <00:35:15.210>Sure. <00:35:15.384>So <00:35:15.560>yeah, <00:35:15.734>I <00:35:15.910>don't <00:35:16.085>see <00:35:16.259>why <00:35:16.435>not.

715
00:35:18.325 --> 00:35:22.655
<00:35:18.325>Perfect. <00:35:18.558>And <00:35:18.791>then <00:35:19.024>the <00:35:19.258>first <00:35:19.491>real <00:35:19.725>question

716
00:35:23.395 --> 00:35:27.495
<00:35:23.395>is, <00:35:23.570>is <00:35:23.745>this <00:35:23.920>state, <00:35:24.095>uh, <00:35:24.270>store <00:35:24.445>transaction <00:35:24.620>is <00:35:24.795>safe

717
00:35:27.755 --> 00:35:29.095
<00:35:27.755>or <00:35:27.996>how <00:35:28.237>do <00:35:28.478>you <00:35:28.719>manage <00:35:28.961>that?

718
00:35:31.755 --> 00:35:34.095
<00:35:31.755>So <00:35:31.988>the <00:35:32.221>state <00:35:32.455>store <00:35:32.688>writes, <00:35:32.921>um, <00:35:33.155>so

719
00:35:34.605 --> 00:35:36.535
<00:35:34.605>it's <00:35:34.805>basically <00:35:35.005>the <00:35:35.205>idea <00:35:35.405>is <00:35:35.605>that <00:35:35.805>you <00:35:36.005>write

720
00:35:36.595 --> 00:35:38.005
<00:35:36.595>to <00:35:36.912>the <00:35:37.229>local <00:35:37.546>state <00:35:37.864>store

721
00:35:38.265 --> 00:35:41.245
<00:35:38.265>and <00:35:38.545>then <00:35:38.825>the <00:35:39.105>change <00:35:39.384>lock <00:35:39.665>topic,

722
00:35:41.305 --> 00:35:43.925
<00:35:41.305>the <00:35:41.585>Kafka <00:35:41.865>topic <00:35:42.145>is <00:35:42.424>updated <00:35:42.705>asynchronously.

723
00:35:44.105 --> 00:35:48.965
<00:35:44.105>Um, <00:35:44.245>that <00:35:44.385>does <00:35:44.525>mean, <00:35:44.665>I <00:35:44.805>mean, <00:35:44.945>I <00:35:45.085>think <00:35:45.225>it <00:35:45.365>can <00:35:45.505>mean

724
00:35:48.965 --> 00:35:52.365
<00:35:48.965>that <00:35:49.165>a <00:35:49.365>change <00:35:49.565>happens <00:35:49.765>only <00:35:49.965>in <00:35:50.165>the <00:35:50.365>local

725
00:35:53.255 --> 00:35:55.005
<00:35:53.255>state <00:35:53.535>store <00:35:53.815>and <00:35:54.095>then <00:35:54.375>the <00:35:54.655>topic

726
00:35:55.005 --> 00:35:56.485
<00:35:55.005>doesn't <00:35:55.271>can <00:35:55.537>get <00:35:55.804>updated <00:35:56.070>in <00:35:56.337>time.

727
00:35:56.835 --> 00:35:58.405
<00:35:56.835>Like <00:35:57.068>the <00:35:57.301>service <00:35:57.535>could <00:35:57.768>crash <00:35:58.001>in <00:35:58.235>between.

728
00:35:59.465 --> 00:36:01.685
<00:35:59.465>But <00:35:59.698>if <00:35:59.931>the <00:36:00.165>service <00:36:00.398>starts <00:36:00.631>up <00:36:00.865>again,

729
00:36:02.165 --> 00:36:05.725
<00:36:02.165>actually <00:36:02.340>it <00:36:02.515>will <00:36:02.690>reprocess <00:36:02.865>the <00:36:03.040>same <00:36:03.215>input <00:36:03.390>event <00:36:03.565>again

730
00:36:05.725 --> 00:36:08.125
<00:36:05.725>because <00:36:05.958>it <00:36:06.191>hasn't <00:36:06.424>actually <00:36:06.658>finished <00:36:06.891>the <00:36:07.125>processing.

731
00:36:08.905 --> 00:36:11.165
<00:36:08.905>So <00:36:09.060>you <00:36:09.216>will <00:36:09.371>then <00:36:09.527>write <00:36:09.682>to <00:36:09.838>the <00:36:09.993>states <00:36:10.149>store <00:36:10.305>again.

732
00:36:11.165 --> 00:36:14.405
<00:36:11.165>So <00:36:11.340>you <00:36:11.515>whole <00:36:11.690>system <00:36:11.865>is <00:36:12.040>basically <00:36:12.215>at <00:36:12.390>least <00:36:12.565>once

733
00:36:14.585 --> 00:36:15.805
<00:36:14.585>by <00:36:14.859>default <00:36:15.134>in <00:36:15.408>the <00:36:15.683>sense

734
00:36:15.805 --> 00:36:18.605
<00:36:15.805>that <00:36:15.944>you <00:36:16.085>will <00:36:16.225>do <00:36:16.365>the <00:36:16.504>stay <00:36:16.645>store <00:36:16.785>right <00:36:16.924>at <00:36:17.065>least <00:36:17.205>once.

735
00:36:18.665 --> 00:36:20.485
<00:36:18.665>And <00:36:18.898>because <00:36:19.131>it's <00:36:19.365>a <00:36:19.598>key <00:36:19.831>value <00:36:20.065>story,

736
00:36:20.485 --> 00:36:22.005
<00:36:20.485>you're <00:36:20.827>basically <00:36:21.169>just <00:36:21.511>overriding <00:36:21.853>the

737
00:36:22.005 --> 00:36:23.045
<00:36:22.005>value <00:36:22.239>with <00:36:22.473>the <00:36:22.707>same <00:36:22.941>value.

738
00:36:23.225 --> 00:36:25.445
<00:36:23.225>So <00:36:23.424>there's <00:36:23.625>not <00:36:23.825>much <00:36:24.025>of <00:36:24.225>a <00:36:24.424>problem <00:36:24.625>here.

739
00:36:26.985 --> 00:36:30.805
<00:36:26.985>Mm-hmm. <00:36:27.335>Um, <00:36:27.685>what <00:36:28.035>is <00:36:28.385>your

740
00:36:31.395 --> 00:36:35.725
<00:36:31.395>opinion <00:36:31.595>on <00:36:31.795>microservices <00:36:31.995>that <00:36:32.195>keep <00:36:32.395>the <00:36:32.595>data <00:36:32.795>normalized

741
00:36:35.905 --> 00:36:39.565
<00:36:35.905>and <00:36:36.138>then <00:36:36.371>provide <00:36:36.605>it <00:36:36.838>as <00:36:37.071>A-C-R-U-D <00:36:37.305>service?

742
00:36:40.265 --> 00:36:41.285
<00:36:40.265>So <00:36:40.723>API <00:36:41.183>based

743
00:36:41.385 --> 00:36:43.565
<00:36:41.385>or <00:36:42.085>orchestration <00:36:42.785>architecture

744
00:36:44.175 --> 00:36:47.565
<00:36:44.175>isn't <00:36:44.350>this <00:36:44.525>often <00:36:44.700>the <00:36:44.875>first <00:36:45.050>typical <00:36:45.225>conceptual <00:36:45.400>error <00:36:45.575>when

745
00:36:47.915 --> 00:36:50.765
<00:36:47.915>designing <00:36:48.381>the <00:36:48.848>microservice <00:36:49.315>architecture?

746
00:36:55.185 --> 00:36:56.405
<00:36:55.185>Uh, <00:36:55.404>I <00:36:55.624>missed <00:36:55.843>the <00:36:56.063>first <00:36:56.283>part

747
00:36:56.405 --> 00:36:57.685
<00:36:56.405>of <00:36:56.597>the <00:36:56.789>question, <00:36:56.981>sorry. <00:36:57.173>But <00:36:57.365>did <00:36:57.557>you,

748
00:36:57.865 --> 00:37:00.485
<00:36:57.865>The <00:36:58.020>first <00:36:58.176>part <00:36:58.331>is <00:36:58.487>what <00:36:58.642>is <00:36:58.798>your <00:36:58.953>opinion <00:36:59.109>on <00:36:59.265>microservices

749
00:37:00.515 --> 00:37:02.565
<00:37:00.515>that <00:37:00.748>keep <00:37:00.981>the <00:37:01.214>data <00:37:01.448>normalized <00:37:01.681>and <00:37:01.915>then

750
00:37:02.925 --> 00:37:04.165
<00:37:02.925>provided <00:37:03.204>as <00:37:03.483>a <00:37:03.762>crowd <00:37:04.041>service?

751
00:37:05.805 --> 00:37:09.565
<00:37:05.805>I, <00:37:06.154>Um, <00:37:06.504>I <00:37:06.855>mean <00:37:07.205>I,

752
00:37:09.805 --> 00:37:11.685
<00:37:09.805>I <00:37:10.038>don't <00:37:10.271>see <00:37:10.504>any <00:37:10.738>problem <00:37:10.971>with <00:37:11.205>that.

753
00:37:11.925 --> 00:37:13.245
<00:37:11.925>I, <00:37:12.123>I <00:37:12.321>think <00:37:12.519>it's <00:37:12.717>a <00:37:12.915>valid <00:37:13.113>approach.

754
00:37:13.785 --> 00:37:16.565
<00:37:13.785>Um, <00:37:13.984>but <00:37:14.185>uh, <00:37:14.384>if, <00:37:14.585>if <00:37:14.785>we're <00:37:14.984>talking <00:37:15.185>about

755
00:37:16.665 --> 00:37:20.445
<00:37:16.665>how <00:37:16.781>it <00:37:16.898>fits <00:37:17.015>into <00:37:17.131>this <00:37:17.248>model, <00:37:17.365>uh, <00:37:17.481>you <00:37:17.598>will <00:37:17.715>have <00:37:17.831>to <00:37:17.948>then <00:37:18.065>go

756
00:37:20.445 --> 00:37:24.605
<00:37:20.445>through <00:37:20.620>some <00:37:20.795>hoops <00:37:20.970>to <00:37:21.145>actually <00:37:21.320>integrate, <00:37:21.495>uh, <00:37:21.670>that <00:37:21.845>with,

757
00:37:24.605 --> 00:37:25.645
<00:37:24.605>with <00:37:25.073>Kafka <00:37:25.541>streams.

758
00:37:26.215 --> 00:37:29.285
<00:37:26.215>Again, <00:37:26.415>two <00:37:26.615>very <00:37:26.815>different <00:37:27.015>approaches, <00:37:27.215>both <00:37:27.415>very <00:37:27.615>valid.

759
00:37:32.275 --> 00:37:36.805
<00:37:32.275>Okay. <00:37:32.508>I <00:37:32.741>hope <00:37:32.975>this <00:37:33.208>answers <00:37:33.441>Michael's <00:37:33.675>question.

760
00:37:38.705 --> 00:37:42.805
<00:37:38.705>Uh, <00:37:38.985>then <00:37:39.265>Alexander <00:37:39.545>wants <00:37:39.825>to <00:37:40.105>know,

761
00:37:42.895 --> 00:37:44.085
<00:37:42.895>could <00:37:43.109>you <00:37:43.323>say <00:37:43.537>a <00:37:43.751>few <00:37:43.966>words,

762
00:37:44.345 --> 00:37:47.125
<00:37:44.345>how <00:37:44.578>do <00:37:44.811>you <00:37:45.044>test <00:37:45.278>your <00:37:45.511>Kafka <00:37:45.745>streams?

763
00:37:47.705 --> 00:37:50.125
<00:37:47.705>Uh, <00:37:47.985>synchronous <00:37:48.265>events <00:37:48.545>make <00:37:48.825>testing <00:37:49.105>more

764
00:37:50.125 --> 00:37:51.885
<00:37:50.125>difficult, <00:37:50.825>don't <00:37:51.525>they?

765
00:37:54.625 --> 00:37:57.975
<00:37:54.625>Right. <00:37:54.765>So <00:37:54.905>I <00:37:55.045>mean, <00:37:55.185>we <00:37:55.325>do <00:37:55.465>test <00:37:55.605>the <00:37:55.745>single <00:37:55.885>Kafka <00:37:56.025>streams

766
00:37:57.975 --> 00:38:00.095
<00:37:57.975>topologies <00:37:58.174>in <00:37:58.375>a <00:37:58.575>kind <00:37:58.775>of <00:37:58.975>unit <00:37:59.174>testing <00:37:59.375>passion.

767
00:38:00.235 --> 00:38:04.175
<00:38:00.235>So <00:38:00.390>Kafka <00:38:00.546>streams <00:38:00.701>test <00:38:00.857>this, <00:38:01.012>um, <00:38:01.168>test <00:38:01.323>topology <00:38:01.479>driver <00:38:01.635>class

768
00:38:04.175 --> 00:38:08.095
<00:38:04.175>where <00:38:04.330>it <00:38:04.486>can <00:38:04.641>basically <00:38:04.797>insert <00:38:04.952>an <00:38:05.108>input, <00:38:05.263>top <00:38:05.419>input <00:38:05.575>event

769
00:38:08.095 --> 00:38:11.175
<00:38:08.095>and <00:38:08.234>then <00:38:08.375>see <00:38:08.515>that <00:38:08.654>the <00:38:08.794>output <00:38:08.935>event <00:38:09.075>that <00:38:09.214>you <00:38:09.355>expect, <00:38:09.495>uh,

770
00:38:11.515 --> 00:38:12.615
<00:38:11.515>get <00:38:11.712>out <00:38:11.911>on <00:38:12.109>the <00:38:12.307>other <00:38:12.504>side.

771
00:38:12.915 --> 00:38:14.735
<00:38:12.915>But <00:38:13.090>of <00:38:13.265>course, <00:38:13.440>it's <00:38:13.615>more <00:38:13.790>like <00:38:13.965>a <00:38:14.140>local <00:38:14.315>testing

772
00:38:15.285 --> 00:38:16.975
<00:38:15.285>when <00:38:15.518>it <00:38:15.751>comes <00:38:15.984>to <00:38:16.218>all <00:38:16.451>our <00:38:16.685>testing.

773
00:38:17.045 --> 00:38:20.095
<00:38:17.045>What <00:38:17.220>we <00:38:17.395>did <00:38:17.570>so <00:38:17.745>far <00:38:17.920>is <00:38:18.095>do <00:38:18.270>integration <00:38:18.445>tests

774
00:38:20.095 --> 00:38:21.895
<00:38:20.095>through <00:38:20.561>our <00:38:21.028>rest <00:38:21.495>API.

775
00:38:21.915 --> 00:38:24.335
<00:38:21.915>So <00:38:22.090>we <00:38:22.265>would <00:38:22.440>do <00:38:22.615>calls <00:38:22.790>through <00:38:22.965>the <00:38:23.140>rest <00:38:23.315>API

776
00:38:24.335 --> 00:38:27.175
<00:38:24.335>and <00:38:24.490>see <00:38:24.646>that <00:38:24.801>everything <00:38:24.957>happens <00:38:25.112>that <00:38:25.268>is <00:38:25.423>supposed <00:38:25.579>to <00:38:25.735>happen.

777
00:38:27.195 --> 00:38:29.895
<00:38:27.195>And <00:38:27.350>that <00:38:27.506>requires <00:38:27.661>all <00:38:27.817>the <00:38:27.972>servers <00:38:28.128>of <00:38:28.283>course, <00:38:28.439>to <00:38:28.595>collaborate

778
00:38:30.045 --> 00:38:31.935
<00:38:30.045>that <00:38:30.325>are <00:38:30.605>behind <00:38:30.885>the <00:38:31.165>rest <00:38:31.445>API.

779
00:38:32.395 --> 00:38:35.845
<00:38:32.395>So <00:38:32.522>we <00:38:32.649>basically <00:38:32.776>have <00:38:32.904>to <00:38:33.031>local <00:38:33.158>topology <00:38:33.285>tests <00:38:33.413>on <00:38:33.540>the <00:38:33.667>one <00:38:33.795>hand

780
00:38:35.945 --> 00:38:38.845
<00:38:35.945>and <00:38:36.100>the <00:38:36.256>integration <00:38:36.411>test <00:38:36.567>API <00:38:36.722>based <00:38:36.878>on <00:38:37.033>the <00:38:37.189>other <00:38:37.345>hand.

781
00:38:40.265 --> 00:38:43.845
<00:38:40.265>So <00:38:40.404>one, <00:38:40.545>one <00:38:40.685>thing <00:38:40.825>that, <00:38:40.964>uh, <00:38:41.105>we <00:38:41.245>didn't <00:38:41.384>implement <00:38:41.525>yet <00:38:41.665>was

782
00:38:43.905 --> 00:38:47.965
<00:38:43.905>to <00:38:44.045>use <00:38:44.185>some, <00:38:44.325>uh, <00:38:44.465>for <00:38:44.605>example, <00:38:44.745>test <00:38:44.885>data <00:38:45.025>sets <00:38:45.165>from, <00:38:45.305>uh,

783
00:38:47.965 --> 00:38:49.525
<00:38:47.965>staging <00:38:48.431>reproduction <00:38:48.898>to, <00:38:49.365>uh,

784
00:38:50.065 --> 00:38:52.485
<00:38:50.065>see <00:38:50.298>whether <00:38:50.531>we're <00:38:50.765>making <00:38:50.998>any <00:38:51.231>braking <00:38:51.465>changes.

785
00:38:53.285 --> 00:38:55.165
<00:38:53.285>I <00:38:53.460>think <00:38:53.634>that <00:38:53.810>would <00:38:53.984>be <00:38:54.160>an <00:38:54.335>interesting <00:38:54.509>next <00:38:54.685>step

786
00:38:55.165 --> 00:38:58.845
<00:38:55.165>because <00:38:55.292>you <00:38:55.419>have <00:38:55.546>to <00:38:55.674>be <00:38:55.801>very <00:38:55.928>careful <00:38:56.055>on <00:38:56.183>what <00:38:56.310>sort <00:38:56.437>of, <00:38:56.565>uh,

787
00:38:58.845 --> 00:39:00.685
<00:38:58.845>schemas <00:38:59.078>you <00:38:59.311>use <00:38:59.544>for, <00:38:59.778>we <00:39:00.011>used <00:39:00.245>Avro,

788
00:39:00.825 --> 00:39:02.205
<00:39:00.825>for <00:39:01.073>example, <00:39:01.321>in <00:39:01.570>our <00:39:01.818>use <00:39:02.067>case.

789
00:39:02.905 --> 00:39:06.405
<00:39:02.905>And, <00:39:03.021>uh, <00:39:03.138>you <00:39:03.255>just <00:39:03.371>have <00:39:03.488>to <00:39:03.605>be <00:39:03.721>very, <00:39:03.838>very <00:39:03.955>careful <00:39:04.071>on <00:39:04.188>the <00:39:04.305>type

790
00:39:06.405 --> 00:39:07.845
<00:39:06.405>of <00:39:06.621>changes <00:39:06.837>you <00:39:07.053>make <00:39:07.269>to <00:39:07.484>those <00:39:07.701>schemas.

791
00:39:07.845 --> 00:39:10.605
<00:39:07.845>And <00:39:07.984>having <00:39:08.125>a <00:39:08.265>way <00:39:08.404>to <00:39:08.544>automatically <00:39:08.685>check <00:39:08.825>those <00:39:08.964>would <00:39:09.105>be <00:39:09.245>very

792
00:39:10.605 --> 00:39:12.885
<00:39:10.605>beneficial, <00:39:10.780>which <00:39:10.955>we <00:39:11.130>didn't <00:39:11.305>have <00:39:11.480>time <00:39:11.655>to <00:39:11.830>do <00:39:12.005>yet.

793
00:39:13.705 --> 00:39:17.045
<00:39:13.705>Mm-hmm. <00:39:14.055>So <00:39:14.404>the <00:39:14.755>next <00:39:15.105>question

794
00:39:18.585 --> 00:39:20.325
<00:39:18.585>is, <00:39:19.985>yeah.

795
00:39:21.105 --> 00:39:25.045
<00:39:21.105>Um, <00:39:21.455>is <00:39:21.805>from <00:39:22.155>Dennis, <00:39:22.505>uh,

796
00:39:25.665 --> 00:39:28.965
<00:39:25.665>all <00:39:25.840>together, <00:39:26.015>this <00:39:26.190>seems <00:39:26.365>to <00:39:26.540>be <00:39:26.715>a <00:39:26.890>little <00:39:27.065>over-engineered,

797
00:39:29.715 --> 00:39:33.405
<00:39:29.715>even <00:39:29.855>though <00:39:29.995>I <00:39:30.135>can <00:39:30.275>understand <00:39:30.415>it <00:39:30.555>is <00:39:30.695>fun <00:39:30.835>working <00:39:30.975>on <00:39:31.115>it.

798
00:39:34.155 --> 00:39:36.765
<00:39:34.155>What <00:39:34.388>do <00:39:34.621>you <00:39:34.855>think <00:39:35.088>about <00:39:35.321>this? <00:39:35.555>Honestly,

799
00:39:37.515 --> 00:39:40.765
<00:39:37.515>Mark, <00:39:37.690>I <00:39:37.865>think <00:39:38.040>there's <00:39:38.214>a <00:39:38.390>grain <00:39:38.565>of <00:39:38.740>truth, <00:39:38.915>definitely.

800
00:39:40.765 --> 00:39:43.685
<00:39:40.765>Yeah. <00:39:40.940>So <00:39:41.115>I <00:39:41.290>would <00:39:41.464>say <00:39:41.640>that <00:39:41.815>this <00:39:41.990>is <00:39:42.165>overkill

801
00:39:43.705 --> 00:39:44.885
<00:39:43.705>for <00:39:44.236>many <00:39:44.767>scenarios.

802
00:39:45.575 --> 00:39:48.325
<00:39:45.575>We're <00:39:45.714>not <00:39:45.855>advocating <00:39:45.995>that <00:39:46.134>you <00:39:46.274>now <00:39:46.415>go <00:39:46.555>out <00:39:46.694>and <00:39:46.835>do <00:39:46.975>this.

803
00:39:48.705 --> 00:39:50.845
<00:39:48.705>That's <00:39:48.904>why <00:39:49.105>we're <00:39:49.305>also <00:39:49.505>trying <00:39:49.705>to <00:39:49.904>make <00:39:50.105>our,

804
00:39:50.905 --> 00:39:52.765
<00:39:50.905>our <00:39:51.185>lessons <00:39:51.465>learned <00:39:51.745>quite <00:39:52.025>clear <00:39:52.305>and,

805
00:39:52.765 --> 00:39:54.685
<00:39:52.765>and, <00:39:52.964>uh, <00:39:53.165>the <00:39:53.365>pain <00:39:53.565>points <00:39:53.765>that <00:39:53.964>we <00:39:54.165>experienced.

806
00:39:55.545 --> 00:39:59.485
<00:39:55.545>Um, <00:39:55.700>but, <00:39:55.856>uh, <00:39:56.011>I <00:39:56.167>think <00:39:56.322>with <00:39:56.478>these <00:39:56.633>hybrid <00:39:56.789>approaches, <00:39:56.945>uh,

807
00:39:59.545 --> 00:40:02.165
<00:39:59.545>you <00:39:59.700>can <00:39:59.856>gain <00:40:00.011>a <00:40:00.167>lot <00:40:00.322>of <00:40:00.478>power <00:40:00.633>from <00:40:00.789>this <00:40:00.945>system.

808
00:40:02.905 --> 00:40:04.685
<00:40:02.905>You <00:40:03.045>just <00:40:03.185>have <00:40:03.325>to <00:40:03.465>make <00:40:03.605>sure <00:40:03.745>that, <00:40:03.885>like, <00:40:04.025>what <00:40:04.165>we <00:40:04.305>try

809
00:40:04.685 --> 00:40:07.405
<00:40:04.685>to <00:40:04.840>do <00:40:04.996>is <00:40:05.151>we <00:40:05.307>try <00:40:05.462>to <00:40:05.618>stay <00:40:05.773>within <00:40:05.929>the <00:40:06.085>realm

810
00:40:07.565 --> 00:40:10.205
<00:40:07.565>of <00:40:07.705>pure <00:40:07.845>Kafka <00:40:07.985>streams <00:40:08.125>as <00:40:08.265>far <00:40:08.405>as <00:40:08.545>we <00:40:08.685>could <00:40:08.825>take <00:40:08.965>it.

811
00:40:10.745 --> 00:40:12.605
<00:40:10.745>Uh, <00:40:10.884>so <00:40:11.025>we <00:40:11.165>really <00:40:11.305>wanted <00:40:11.444>to <00:40:11.585>see <00:40:11.725>what <00:40:11.865>we <00:40:12.005>could <00:40:12.145>do.

812
00:40:13.145 --> 00:40:16.045
<00:40:13.145>Uh, <00:40:13.300>but <00:40:13.456>we <00:40:13.611>did <00:40:13.767>run <00:40:13.922>into <00:40:14.078>these <00:40:14.233>sort <00:40:14.389>of <00:40:14.545>limitations

813
00:40:16.045 --> 00:40:17.125
<00:40:16.045>where <00:40:16.288>we're <00:40:16.531>now <00:40:16.774>looking <00:40:17.017>back

814
00:40:17.125 --> 00:40:19.685
<00:40:17.125>and <00:40:17.265>saying, <00:40:17.405>actually <00:40:17.545>we <00:40:17.685>need <00:40:17.825>to <00:40:17.965>take <00:40:18.105>a <00:40:18.245>step <00:40:18.385>back <00:40:18.525>and,

815
00:40:19.685 --> 00:40:21.605
<00:40:19.685>and <00:40:19.825>think <00:40:19.965>of <00:40:20.105>a <00:40:20.245>different <00:40:20.384>approach <00:40:20.525>for <00:40:20.665>some <00:40:20.805>of <00:40:20.945>these <00:40:21.085>parts.

816
00:40:21.875 --> 00:40:23.965
<00:40:21.875>Exactly. <00:40:22.030>It <00:40:22.186>was <00:40:22.341>about <00:40:22.497>how <00:40:22.652>far <00:40:22.808>can <00:40:22.963>we <00:40:23.119>push <00:40:23.275>it

817
00:40:24.105 --> 00:40:26.605
<00:40:24.105>and <00:40:24.338>yeah, <00:40:24.571>we, <00:40:24.805>we <00:40:25.038>saw <00:40:25.271>the <00:40:25.505>problem.

818
00:40:26.865 --> 00:40:28.085
<00:40:26.865>So <00:40:27.139>we're <00:40:27.413>trying <00:40:27.688>to <00:40:27.963>like,

819
00:40:28.085 --> 00:40:30.125
<00:40:28.085>and <00:40:28.285>with <00:40:28.485>this <00:40:28.685>here <00:40:28.885>trying <00:40:29.085>to <00:40:29.285>see, <00:40:29.485>okay,

820
00:40:30.125 --> 00:40:31.605
<00:40:30.125>let's <00:40:30.458>try <00:40:30.791>more <00:40:31.124>hybrid <00:40:31.457>approach.

821
00:40:31.655 --> 00:40:34.805
<00:40:31.655>Let's <00:40:31.888>try <00:40:32.121>to <00:40:32.355>actually <00:40:32.588>use <00:40:32.821>other <00:40:33.055>approaches

822
00:40:34.805 --> 00:40:36.765
<00:40:34.805>where <00:40:34.960>it <00:40:35.116>makes <00:40:35.271>sense <00:40:35.427>and <00:40:35.582>find <00:40:35.738>the <00:40:35.893>right <00:40:36.049>middle <00:40:36.205>ground.

823
00:40:37.065 --> 00:40:40.245
<00:40:37.065>But <00:40:37.220>that <00:40:37.376>required <00:40:37.531>us <00:40:37.687>to <00:40:37.842>first <00:40:37.998>go <00:40:38.153>the <00:40:38.309>extreme <00:40:38.465>way

824
00:40:42.025 --> 00:40:46.645
<00:40:42.025>Is, <00:40:42.200>uh, <00:40:42.375>but <00:40:42.550>like <00:40:42.725>there <00:40:42.900>is <00:40:43.075>an <00:40:43.250>extreme, <00:40:43.425>yeah,

825
00:40:46.815 --> 00:40:48.645
<00:40:46.815>let's <00:40:47.095>call <00:40:47.375>it, <00:40:47.655>uh, <00:40:47.935>Kafka <00:40:48.215>one.

826
00:40:48.825 --> 00:40:51.605
<00:40:48.825>And <00:40:48.980>then, <00:40:49.136>uh, <00:40:49.291>on <00:40:49.447>the <00:40:49.602>other <00:40:49.758>end <00:40:49.913>there's <00:40:50.069>Kafka <00:40:50.225>zero.

827
00:40:51.905 --> 00:40:56.325
<00:40:51.905>But <00:40:52.032>how <00:40:52.159>do <00:40:52.286>you, <00:40:52.414>how, <00:40:52.541>how <00:40:52.668>would <00:40:52.795>you, <00:40:52.923>how <00:40:53.050>would <00:40:53.177>you <00:40:53.305>realize

828
00:40:56.915 --> 00:40:59.605
<00:40:56.915>that <00:40:57.148>the <00:40:57.381>amount <00:40:57.615>of <00:40:57.848>Kafka <00:40:58.081>you <00:40:58.315>have

829
00:41:00.705 --> 00:41:04.765
<00:41:00.705>is <00:41:00.880>good <00:41:01.055>enough <00:41:01.230>for <00:41:01.404>what <00:41:01.580>you <00:41:01.755>want <00:41:01.930>to <00:41:02.105>do?

830
00:41:08.425 --> 00:41:09.565
<00:41:08.425>That's <00:41:08.767>an <00:41:09.109>interesting <00:41:09.451>question.

831
00:41:09.905 --> 00:41:12.045
<00:41:09.905>Um, <00:41:10.138>I <00:41:10.371>mean, <00:41:10.605>there <00:41:10.838>are <00:41:11.071>some <00:41:11.305>principles

832
00:41:12.155 --> 00:41:15.205
<00:41:12.155>that <00:41:12.330>we <00:41:12.505>have <00:41:12.680>on <00:41:12.855>the <00:41:13.030>architecture <00:41:13.205>without <00:41:13.380>which <00:41:13.555>we

833
00:41:15.205 --> 00:41:16.405
<00:41:15.205>don't <00:41:15.421>get <00:41:15.637>the <00:41:15.853>benefits <00:41:16.069>we <00:41:16.285>want.

834
00:41:16.875 --> 00:41:18.125
<00:41:16.875>Like, <00:41:17.156>for <00:41:17.437>instance, <00:41:17.718>the <00:41:18.000>idea

835
00:41:18.125 --> 00:41:21.365
<00:41:18.125>of <00:41:18.252>having <00:41:18.379>all <00:41:18.506>the <00:41:18.634>data <00:41:18.761>available <00:41:18.888>in <00:41:19.015>cover, <00:41:19.143>so <00:41:19.270>that <00:41:19.397>can <00:41:19.525>be

836
00:41:22.085 --> 00:41:23.045
<00:41:22.085>combined <00:41:22.301>and <00:41:22.517>mix <00:41:22.733>and <00:41:22.949>match

837
00:41:23.075 --> 00:41:24.525
<00:41:23.075>reacted <00:41:23.336>to <00:41:23.596>from <00:41:23.858>any <00:41:24.119>other <00:41:24.380>service.

838
00:41:24.985 --> 00:41:27.605
<00:41:24.985>So <00:41:25.112>that's <00:41:25.239>something <00:41:25.366>that <00:41:25.494>we <00:41:25.621>don't <00:41:25.748>want <00:41:25.875>to <00:41:26.003>go <00:41:26.130>go <00:41:26.257>away <00:41:26.385>from.

839
00:41:28.065 --> 00:41:30.685
<00:41:28.065>But <00:41:28.205>that <00:41:28.345>doesn't <00:41:28.485>have <00:41:28.625>to <00:41:28.765>mean <00:41:28.905>that <00:41:29.045>all <00:41:29.185>that <00:41:29.325>Kafka <00:41:29.465>has

840
00:41:30.685 --> 00:41:32.205
<00:41:30.685>to <00:41:30.913>be <00:41:31.141>the <00:41:31.369>primary <00:41:31.596>form <00:41:31.825>of <00:41:32.053>storage.

841
00:41:32.715 --> 00:41:35.405
<00:41:32.715>Like <00:41:32.855>with <00:41:32.995>an <00:41:33.135>approach <00:41:33.275>like <00:41:33.415>this <00:41:33.555>one <00:41:33.695>with <00:41:33.835>the <00:41:33.975>database <00:41:34.115>in

842
00:41:35.405 --> 00:41:37.325
<00:41:35.405>between <00:41:35.871>probably <00:41:36.338>the <00:41:36.805>database,

843
00:41:37.325 --> 00:41:39.325
<00:41:37.325>what <00:41:37.524>we <00:41:37.725>would <00:41:37.924>consider <00:41:38.125>the <00:41:38.325>source <00:41:38.524>of <00:41:38.725>truth,

844
00:41:39.385 --> 00:41:41.245
<00:41:39.385>and <00:41:39.560>then <00:41:39.735>we <00:41:39.910>would <00:41:40.085>just <00:41:40.260>publish <00:41:40.435>it <00:41:40.610>on <00:41:40.785>Kafka.

845
00:41:41.505 --> 00:41:45.485
<00:41:41.505>But <00:41:41.645>we <00:41:41.785>still <00:41:41.925>preserve <00:41:42.065>the <00:41:42.205>benefit <00:41:42.345>that <00:41:42.485>all <00:41:42.625>data <00:41:42.765>is <00:41:42.905>available

846
00:41:45.905 --> 00:41:47.405
<00:41:45.905>for <00:41:46.355>collaboration <00:41:46.805>in <00:41:47.255>Kafka.

847
00:41:47.425 --> 00:41:52.285
<00:41:47.425>So <00:41:47.600>basically <00:41:47.775>that <00:41:47.950>we <00:41:48.125>want <00:41:48.300>to <00:41:48.475>maintain <00:41:48.650>specific <00:41:48.825>goals

848
00:41:53.025 --> 00:41:54.285
<00:41:53.025>and <00:41:53.403>we <00:41:53.781>always <00:41:54.158>have

849
00:41:54.285 --> 00:41:56.365
<00:41:54.285>to <00:41:54.424>see <00:41:54.565>when <00:41:54.705>we <00:41:54.845>go <00:41:54.984>a <00:41:55.125>step <00:41:55.265>back <00:41:55.404>from <00:41:55.545>pure <00:41:55.685>Kafka,

850
00:41:56.365 --> 00:41:59.845
<00:41:56.365>are <00:41:56.504>we <00:41:56.645>still <00:41:56.785>meeting <00:41:56.924>this <00:41:57.064>goals <00:41:57.205>or <00:41:57.345>are <00:41:57.484>goals <00:41:57.625>still <00:41:57.765>relevant?

851
00:42:00.045 --> 00:42:01.725
<00:42:00.045>I <00:42:00.245>think <00:42:00.445>that's <00:42:00.645>the <00:42:00.845>approach <00:42:01.045>we're <00:42:01.245>basically <00:42:01.445>taking,

852
00:42:01.935 --> 00:42:03.245
<00:42:01.935>going <00:42:02.229>back <00:42:02.524>step <00:42:02.819>by <00:42:03.114>step.

853
00:42:03.835 --> 00:42:06.845
<00:42:03.835>Yeah. <00:42:03.951>And <00:42:04.068>one <00:42:04.185>thing <00:42:04.301>that <00:42:04.418>we <00:42:04.535>wanted <00:42:04.651>to <00:42:04.768>do <00:42:04.885>was <00:42:05.001>we, <00:42:05.118>we <00:42:05.235>wanted

854
00:42:06.845 --> 00:42:09.005
<00:42:06.845>to <00:42:07.125>have <00:42:07.404>a <00:42:07.685>system <00:42:07.964>where, <00:42:08.245>uh,

855
00:42:09.005 --> 00:42:11.005
<00:42:09.005>that <00:42:09.355>had <00:42:09.705>very <00:42:10.055>well-defined <00:42:10.405>behavior.

856
00:42:11.225 --> 00:42:13.765
<00:42:11.225>And <00:42:11.341>I <00:42:11.458>think <00:42:11.575>that <00:42:11.691>you <00:42:11.808>can <00:42:11.924>achieve <00:42:12.041>when, <00:42:12.158>when <00:42:12.275>you <00:42:12.391>do <00:42:12.508>a <00:42:12.625>lot

857
00:42:13.765 --> 00:42:17.485
<00:42:13.765>of <00:42:13.940>things <00:42:14.115>with <00:42:14.290>Kafka, <00:42:14.464>um, <00:42:14.640>for <00:42:14.815>example <00:42:14.990>around <00:42:15.165>robustness,

858
00:42:17.865 --> 00:42:19.885
<00:42:17.865>if <00:42:18.064>you <00:42:18.265>introduce <00:42:18.464>a <00:42:18.665>change <00:42:18.865>into <00:42:19.064>the <00:42:19.265>system

859
00:42:19.995 --> 00:42:21.605
<00:42:19.995>that <00:42:20.345>the <00:42:20.694>system <00:42:21.045>has <00:42:21.395>accepted,

860
00:42:22.225 --> 00:42:25.405
<00:42:22.225>you <00:42:22.424>are <00:42:22.625>basically <00:42:22.825>guaranteed <00:42:23.025>if <00:42:23.225>you're <00:42:23.424>configured <00:42:23.625>everything

861
00:42:25.405 --> 00:42:27.405
<00:42:25.405>correctly, <00:42:25.638>that <00:42:25.871>it <00:42:26.105>will <00:42:26.338>propagate <00:42:26.571>over <00:42:26.805>time,

862
00:42:27.405 --> 00:42:30.765
<00:42:27.405>eventually <00:42:27.560>everywhere <00:42:27.716>as <00:42:27.871>need, <00:42:28.027>uh, <00:42:28.182>as <00:42:28.338>it <00:42:28.493>needs <00:42:28.649>to <00:42:28.805>be.

863
00:42:31.385 --> 00:42:34.605
<00:42:31.385>Um, <00:42:31.512>so <00:42:31.639>I, <00:42:31.766>I <00:42:31.894>think <00:42:32.021>that's <00:42:32.148>a <00:42:32.275>very <00:42:32.403>powerful <00:42:32.530>feature <00:42:32.657>in <00:42:32.785>it,

864
00:42:35.225 --> 00:42:37.805
<00:42:35.225>but <00:42:35.352>of <00:42:35.479>course <00:42:35.606>you, <00:42:35.734>you <00:42:35.861>pay <00:42:35.988>upfront <00:42:36.115>a <00:42:36.243>lot <00:42:36.370>to <00:42:36.497>get <00:42:36.625>it.

865
00:42:38.715 --> 00:42:41.165
<00:42:38.715>Yeah. <00:42:39.181>Okay. <00:42:39.648>Thank <00:42:40.115>you.

866
00:42:41.745 --> 00:42:43.325
<00:42:41.745>Uh, <00:42:41.920>by <00:42:42.095>the <00:42:42.270>way, <00:42:42.444>this <00:42:42.620>was <00:42:42.795>a <00:42:42.970>question <00:42:43.145>from

867
00:42:43.325 --> 00:42:44.445
<00:42:43.325>me, <00:42:43.493>not <00:42:43.661>from <00:42:43.829>the <00:42:43.997>q <00:42:44.165>and <00:42:44.333>a.

868
00:42:44.825 --> 00:42:45.885
<00:42:44.825>Um, <00:42:45.779>so,

869
00:42:46.065 --> 00:42:49.485
<00:42:46.065>and <00:42:46.220>then <00:42:46.376>we <00:42:46.531>have, <00:42:46.687>uh, <00:42:46.842>a <00:42:46.998>question <00:42:47.153>from <00:42:47.309>frac, <00:42:47.465>uh,

870
00:42:49.655 --> 00:42:51.725
<00:42:49.655>which <00:42:49.888>I <00:42:50.121>also <00:42:50.355>think <00:42:50.588>is <00:42:50.821>very <00:42:51.055>interesting.

871
00:42:52.065 --> 00:42:55.885
<00:42:52.065>As <00:42:52.205>I <00:42:52.345>understand <00:42:52.485>you <00:42:52.625>keep <00:42:52.765>all <00:42:52.905>the <00:42:53.045>Kafka <00:42:53.185>stream <00:42:53.325>data <00:42:53.465>forever,

872
00:42:57.065 --> 00:43:00.285
<00:42:57.065>is <00:42:57.298>the <00:42:57.531>amount <00:42:57.765>of <00:42:57.998>data <00:42:58.231>low <00:42:58.465>enough?

873
00:43:01.065 --> 00:43:03.485
<00:43:01.065>Um, <00:43:01.265>where <00:43:01.465>do <00:43:01.665>you <00:43:01.865>store <00:43:02.065>all <00:43:02.265>the <00:43:02.465>data?

874
00:43:03.745 --> 00:43:05.565
<00:43:03.745>So <00:43:04.095>hot <00:43:04.444>storage, <00:43:04.795>code, <00:43:05.145>storage,

875
00:43:07.355 --> 00:43:08.355
<00:43:07.355>Storage. <00:43:07.805>So <00:43:08.255>May,

876
00:43:08.355 --> 00:43:10.645
<00:43:08.355>first <00:43:08.555>of <00:43:08.755>all, <00:43:08.955>Kafka <00:43:09.155>also <00:43:09.355>has <00:43:09.555>the <00:43:09.755>concept

877
00:43:10.865 --> 00:43:12.285
<00:43:10.865>of <00:43:11.504>compacted <00:43:12.143>topics.

878
00:43:13.105 --> 00:43:14.405
<00:43:13.105>Um, <00:43:13.495>the <00:43:13.885>concept <00:43:14.275>being

879
00:43:14.405 --> 00:43:17.605
<00:43:14.405>that <00:43:14.560>you <00:43:14.716>only <00:43:14.871>keep <00:43:15.027>the <00:43:15.182>last <00:43:15.338>message <00:43:15.493>for <00:43:15.649>each <00:43:15.805>key.

880
00:43:17.825 --> 00:43:19.085
<00:43:17.825>That's <00:43:18.051>for <00:43:18.278>instance, <00:43:18.505>what <00:43:18.732>we're <00:43:18.959>using

881
00:43:19.185 --> 00:43:20.325
<00:43:19.185>for <00:43:19.527>these <00:43:19.868>entity <00:43:20.211>topics.

882
00:43:20.425 --> 00:43:22.885
<00:43:20.425>So <00:43:20.658>for <00:43:20.891>instance, <00:43:21.125>this <00:43:21.358>inventory <00:43:21.591>items <00:43:21.825>topic

883
00:43:22.945 --> 00:43:25.325
<00:43:22.945>or <00:43:23.120>we <00:43:23.295>just <00:43:23.470>need <00:43:23.645>current, <00:43:23.820>the <00:43:23.995>current <00:43:24.170>inventory <00:43:24.345>item.

884
00:43:25.325 --> 00:43:29.685
<00:43:25.325>Basically <00:43:25.480>we <00:43:25.636>set <00:43:25.791>this <00:43:25.947>up <00:43:26.102>as <00:43:26.258>compacted, <00:43:26.413>so <00:43:26.569>previous <00:43:26.725>states

885
00:43:30.065 --> 00:43:32.085
<00:43:30.065>of <00:43:30.345>this <00:43:30.625>inventory <00:43:30.905>item <00:43:31.185>get <00:43:31.465>eventually

886
00:43:32.085 --> 00:43:33.285
<00:43:32.085>garbage <00:43:32.625>collected, <00:43:33.165>right?

887
00:43:33.545 --> 00:43:35.085
<00:43:33.545>That's <00:43:33.822>one <00:43:34.099>part <00:43:34.376>of <00:43:34.653>the <00:43:34.931>answer.

888
00:43:35.705 --> 00:43:37.565
<00:43:35.705>But <00:43:35.938>of <00:43:36.171>course, <00:43:36.404>with <00:43:36.638>these <00:43:36.871>invent <00:43:37.105>topics,

889
00:43:37.705 --> 00:43:38.765
<00:43:37.705>we <00:43:37.943>do <00:43:38.182>keep <00:43:38.420>data <00:43:38.658>forever.

890
00:43:39.545 --> 00:43:42.845
<00:43:39.545>And <00:43:39.895>luckily <00:43:40.245>nowadays, <00:43:40.595>um, <00:43:40.945>Kafka

891
00:43:42.885 --> 00:43:45.765
<00:43:42.885>and <00:43:43.040>Confluent <00:43:43.196>Cloud <00:43:43.351>is <00:43:43.507>very <00:43:43.662>efficient <00:43:43.818>at <00:43:43.973>storing <00:43:44.129>these <00:43:44.285>very,

892
00:43:46.935 --> 00:43:50.045
<00:43:46.935>these <00:43:47.075>very <00:43:47.215>big <00:43:47.355>topics <00:43:47.495>that <00:43:47.634>go <00:43:47.775>far <00:43:47.915>back <00:43:48.055>into <00:43:48.195>the <00:43:48.335>past.

893
00:43:50.195 --> 00:43:51.885
<00:43:50.195>What, <00:43:50.475>for <00:43:50.755>instance, <00:43:51.035>confluent <00:43:51.315>Cloud <00:43:51.595>does.

894
00:43:51.885 --> 00:43:53.605
<00:43:51.885>What <00:43:52.351>we <00:43:52.818>use <00:43:53.285>is

895
00:43:53.605 --> 00:43:57.765
<00:43:53.605>that <00:43:53.780>very <00:43:53.955>old <00:43:54.130>data <00:43:54.305>automatically <00:43:54.480>archives <00:43:54.655>it <00:43:54.830>in <00:43:55.005>S3

896
00:43:57.825 --> 00:43:59.805
<00:43:57.825>or <00:43:58.058>some <00:43:58.291>other <00:43:58.524>more <00:43:58.758>code <00:43:58.991>storage, <00:43:59.225>right?

897
00:43:59.865 --> 00:44:01.805
<00:43:59.865>So <00:44:00.098>you <00:44:00.331>don't <00:44:00.564>have <00:44:00.798>to <00:44:01.031>think <00:44:01.265>about

898
00:44:01.805 --> 00:44:03.045
<00:44:01.805>as <00:44:02.176>an <00:44:02.549>application <00:44:02.921>developer,

899
00:44:03.065 --> 00:44:06.125
<00:44:03.065>but <00:44:03.205>if <00:44:03.345>you <00:44:03.485>have <00:44:03.625>all <00:44:03.765>data <00:44:03.905>that <00:44:04.045>you <00:44:04.185>very <00:44:04.325>rarely <00:44:04.465>access,

900
00:44:06.125 --> 00:44:08.165
<00:44:06.125>it <00:44:06.405>will <00:44:06.685>be <00:44:06.965>basically <00:44:07.245>moved <00:44:07.525>out

901
00:44:08.165 --> 00:44:10.605
<00:44:08.165>of <00:44:08.445>Kafka <00:44:08.725>itself <00:44:09.005>into <00:44:09.285>quota <00:44:09.565>storage.

902
00:44:11.505 --> 00:44:13.245
<00:44:11.505>And <00:44:11.705>I <00:44:11.905>think <00:44:12.105>you <00:44:12.305>need <00:44:12.505>to <00:44:12.705>really, <00:44:12.905>uh,

903
00:44:13.245 --> 00:44:16.205
<00:44:13.245>look <00:44:13.384>at <00:44:13.525>your <00:44:13.665>own <00:44:13.805>use <00:44:13.944>cases <00:44:14.085>and <00:44:14.225>choose <00:44:14.365>the <00:44:14.505>right <00:44:14.645>approach.

904
00:44:16.625 --> 00:44:20.405
<00:44:16.625>So, <00:44:16.765>um, <00:44:16.905>in <00:44:17.045>our <00:44:17.185>cases, <00:44:17.325>in <00:44:17.465>these <00:44:17.605>examples <00:44:17.745>that <00:44:17.885>we <00:44:18.025>showed,

905
00:44:20.415 --> 00:44:22.925
<00:44:20.415>we're <00:44:20.542>not <00:44:20.669>dealing <00:44:20.796>with <00:44:20.924>so <00:44:21.051>much <00:44:21.178>data <00:44:21.305>that <00:44:21.433>we <00:44:21.560>can't <00:44:21.687>handle <00:44:21.815>it.

906
00:44:22.925 --> 00:44:26.965
<00:44:22.925>But <00:44:23.080>if <00:44:23.236>we <00:44:23.391>started <00:44:23.547>consuming <00:44:23.702>some <00:44:23.858>very <00:44:24.013>high <00:44:24.169>frequency <00:44:24.325>sources

907
00:44:27.105 --> 00:44:30.645
<00:44:27.105>of <00:44:27.232>data, <00:44:27.359>then <00:44:27.486>of <00:44:27.614>course <00:44:27.741>we <00:44:27.868>would <00:44:27.995>not <00:44:28.123>store <00:44:28.250>those <00:44:28.377>as, <00:44:28.505>uh,

908
00:44:30.645 --> 00:44:34.085
<00:44:30.645>events <00:44:30.785>and <00:44:30.925>an <00:44:31.065>event <00:44:31.205>source <00:44:31.345>them <00:44:31.485>that <00:44:31.625>that <00:44:31.765>would <00:44:31.905>be, <00:44:32.045>um,

909
00:44:34.635 --> 00:44:36.325
<00:44:34.635>yeah, <00:44:34.868>that <00:44:35.101>just <00:44:35.335>wouldn't <00:44:35.568>work <00:44:35.801>long <00:44:36.035>term.

910
00:44:36.785 --> 00:44:38.765
<00:44:36.785>Uh, <00:44:36.984>but <00:44:37.185>also <00:44:37.384>we <00:44:37.585>were <00:44:37.785>thinking <00:44:37.984>about <00:44:38.185>what

911
00:44:38.765 --> 00:44:41.285
<00:44:38.765>to <00:44:38.892>do <00:44:39.019>if <00:44:39.146>we <00:44:39.274>actually <00:44:39.401>have <00:44:39.528>made <00:44:39.655>a <00:44:39.783>mistake <00:44:39.910>in <00:44:40.037>the <00:44:40.165>beginning

912
00:44:41.305 --> 00:44:43.445
<00:44:41.305>and, <00:44:41.504>and <00:44:41.705>started <00:44:41.904>to <00:44:42.105>collect <00:44:42.305>a <00:44:42.504>data <00:44:42.705>source

913
00:44:43.475 --> 00:44:45.525
<00:44:43.475>that <00:44:43.674>grows <00:44:43.875>too <00:44:44.075>large, <00:44:44.275>what <00:44:44.475>do <00:44:44.674>we <00:44:44.875>do?

914
00:44:45.825 --> 00:44:50.205
<00:44:45.825>We <00:44:45.980>can <00:44:46.136>always <00:44:46.291>migrate <00:44:46.447>that <00:44:46.602>topic <00:44:46.758>by <00:44:46.913>compacting, <00:44:47.069>for <00:44:47.225>example,

915
00:44:50.665 --> 00:44:53.965
<00:44:50.665>uh, <00:44:50.865>events <00:44:51.065>across <00:44:51.265>or, <00:44:51.465>or <00:44:51.665>for <00:44:51.865>single <00:44:52.065>entities.

916
00:44:54.025 --> 00:44:57.325
<00:44:54.025>So <00:44:54.165>if <00:44:54.305>you <00:44:54.445>have <00:44:54.585>a <00:44:54.725>create <00:44:54.865>update, <00:44:55.005>update <00:44:55.145>update, <00:44:55.285>uh, <00:44:55.425>events,

917
00:44:57.515 --> 00:45:00.085
<00:44:57.515>then <00:44:57.690>we <00:44:57.865>can <00:44:58.040>migrate <00:44:58.214>them <00:44:58.390>to <00:44:58.565>a <00:44:58.740>separate <00:44:58.915>one

918
00:45:00.085 --> 00:45:01.965
<00:45:00.085>where <00:45:00.318>we <00:45:00.551>basically <00:45:00.785>collapse <00:45:01.018>most <00:45:01.251>of <00:45:01.485>those

919
00:45:02.065 --> 00:45:04.565
<00:45:02.065>to <00:45:02.220>perhaps <00:45:02.376>just <00:45:02.531>one <00:45:02.687>created <00:45:02.842>event <00:45:02.998>if <00:45:03.153>we're <00:45:03.309>no <00:45:03.465>longer

920
00:45:04.565 --> 00:45:05.565
<00:45:04.565>interested <00:45:04.865>in <00:45:05.165>that <00:45:05.465>history.

921
00:45:08.715 --> 00:45:13.605
<00:45:08.715>Okay. <00:45:08.915>So, <00:45:09.115>uh, <00:45:09.315>all <00:45:09.515>the <00:45:09.715>questions <00:45:09.915>until <00:45:10.115>now,

922
00:45:14.305 --> 00:45:17.485
<00:45:14.305>um, <00:45:14.654>I <00:45:15.004>think <00:45:15.355>people <00:45:15.705>will

923
00:45:18.035 --> 00:45:22.565
<00:45:18.035>find <00:45:18.174>you <00:45:18.315>on <00:45:18.455>the <00:45:18.595>interwebs, <00:45:18.734>um, <00:45:18.875>if <00:45:19.015>they <00:45:19.154>have <00:45:19.295>more <00:45:19.435>questions.

924
00:45:23.465 --> 00:45:27.165
<00:45:23.465>Um, <00:45:23.815>and <00:45:24.165>I <00:45:24.515>would <00:45:24.865>be

925
00:45:27.675 --> 00:45:31.965
<00:45:27.675>very <00:45:27.875>excited <00:45:28.075>to <00:45:28.275>see <00:45:28.475>where <00:45:28.675>this <00:45:28.875>model <00:45:29.075>gets

926
00:45:32.105 --> 00:45:35.045
<00:45:32.105>to <00:45:32.338>in <00:45:32.571>like <00:45:32.805>one, <00:45:33.038>two <00:45:33.271>years, <00:45:33.505>um,

927
00:45:35.235 --> 00:45:38.845
<00:45:35.235>with <00:45:35.435>Kafka <00:45:35.635>further <00:45:35.835>evolving <00:45:36.035>and <00:45:36.235>stuff <00:45:36.435>like <00:45:36.635>this.

928
00:45:39.905 --> 00:45:43.445
<00:45:39.905>So <00:45:40.045>thank <00:45:40.185>you <00:45:40.325>again <00:45:40.465>for <00:45:40.605>your <00:45:40.745>time <00:45:40.885>and <00:45:41.025>for <00:45:41.165>your <00:45:41.305>talk.