WEBVTT

1
00:00:19.095 --> 00:00:19.445
Hello.

2
00:00:20.025 --> 00:00:22.205
<00:00:20.025>Uh, <00:00:20.258>so <00:00:20.491>my <00:00:20.724>name <00:00:20.958>is <00:00:21.191>Ki <00:00:21.424>Koff.

3
00:00:22.785 --> 00:00:25.445
<00:00:22.785>I'm <00:00:23.018>a <00:00:23.251>consultant <00:00:23.485>engineer <00:00:23.718>with <00:00:23.951>Confluent, <00:00:24.185>uh,

4
00:00:25.505 --> 00:00:28.005
<00:00:25.505>the <00:00:25.705>company <00:00:25.904>founded <00:00:26.105>by, <00:00:26.305>uh, <00:00:26.505>the <00:00:26.705>original <00:00:26.904>creators

5
00:00:28.145 --> 00:00:29.325
<00:00:28.145>of <00:00:28.676>Apache <00:00:29.207>Kafka.

6
00:00:30.185 --> 00:00:34.285
<00:00:30.185>And <00:00:30.340>I'm <00:00:30.496>delighted <00:00:30.651>to <00:00:30.807>be <00:00:30.962>here <00:00:31.118>speaking <00:00:31.273>at <00:00:31.429>Weir <00:00:31.584>developers.

7
00:00:35.785 --> 00:00:38.565
<00:00:35.785>So <00:00:35.985>I <00:00:36.184>work <00:00:36.385>at <00:00:36.584>the <00:00:36.785>professional <00:00:36.985>services <00:00:37.184>team

8
00:00:38.665 --> 00:00:42.565
<00:00:38.665>and <00:00:38.839>our <00:00:39.015>day-to-day <00:00:39.190>job <00:00:39.365>is <00:00:39.540>to <00:00:39.714>help <00:00:39.890>customers <00:00:40.065>succeed

9
00:00:42.565 --> 00:00:43.645
<00:00:42.565>with <00:00:43.051>Apache <00:00:43.537>Kafka.

10
00:00:45.295 --> 00:00:48.275
<00:00:45.295>Apache <00:00:45.470>Kafka <00:00:45.645>is <00:00:45.820>a <00:00:45.995>complex <00:00:46.170>system <00:00:46.345>with <00:00:46.520>multiple <00:00:46.695>parameters

11
00:00:48.295 --> 00:00:50.315
<00:00:48.295>to <00:00:48.575>configure <00:00:48.855>for <00:00:49.135>different <00:00:49.415>use <00:00:49.695>cases.

12
00:00:53.215 --> 00:00:56.115
<00:00:53.215>And, <00:00:53.415>uh, <00:00:53.615>as <00:00:53.815>a <00:00:54.015>consultant <00:00:54.215>engineer <00:00:54.415>in <00:00:54.615>Confluent,

13
00:00:56.275 --> 00:00:59.475
<00:00:56.275>I <00:00:56.430>can <00:00:56.586>see <00:00:56.741>many <00:00:56.897>clients <00:00:57.052>who <00:00:57.208>need <00:00:57.363>to <00:00:57.519>benchmark, <00:00:57.675>uh,

14
00:00:59.475 --> 00:01:02.395
<00:00:59.475>their <00:00:59.708>production <00:00:59.941>systems <00:01:00.175>and <00:01:00.408>understand <00:01:00.641>their <00:01:00.875>capacity.

15
00:01:06.455 --> 00:01:08.515
<00:01:06.455>So <00:01:07.855>benchmarking

16
00:01:08.515 --> 00:01:11.715
<00:01:08.515>and <00:01:08.690>measuring <00:01:08.865>are <00:01:09.040>not <00:01:09.215>just <00:01:09.390>for <00:01:09.565>finding <00:01:09.740>a <00:01:09.915>bottleneck,

17
00:01:12.175 --> 00:01:14.915
<00:01:12.175>you <00:01:12.350>need <00:01:12.524>them <00:01:12.700>to <00:01:12.875>better <00:01:13.050>understand <00:01:13.225>the <00:01:13.399>loads <00:01:13.575>you

18
00:01:14.915 --> 00:01:15.955
<00:01:14.914>are <00:01:15.148>placing <00:01:15.383>on <00:01:15.616>the <00:01:15.851>system.

19
00:01:17.415 --> 00:01:19.995
<00:01:17.414>And <00:01:17.648>benchmark <00:01:17.881>details <00:01:18.115>can <00:01:18.348>be <00:01:18.581>used, <00:01:18.815>uh,

20
00:01:20.055 --> 00:01:22.675
<00:01:20.055>to <00:01:20.521>optimize <00:01:20.988>throughput <00:01:21.455>latency

21
00:01:23.575 --> 00:01:25.275
<00:01:23.575>and <00:01:24.041>better <00:01:24.508>utilized <00:01:24.975>resources.

22
00:01:27.105 --> 00:01:29.125
<00:01:27.105>Uh, <00:01:27.338>this <00:01:27.571>talk <00:01:27.805>will <00:01:28.038>focus <00:01:28.271>on <00:01:28.505>methods

23
00:01:29.305 --> 00:01:31.165
<00:01:29.305>and <00:01:29.585>tools <00:01:29.865>of <00:01:30.145>Apache <00:01:30.425>Kafka <00:01:30.705>performance

24
00:01:31.485 --> 00:01:32.645
<00:01:31.485>analysis <00:01:32.007>and <00:01:32.529>benchmarking.

25
00:01:34.665 --> 00:01:37.765
<00:01:34.664>And <00:01:35.014>uh, <00:01:35.365>today <00:01:35.714>I'm <00:01:36.065>going

26
00:01:37.765 --> 00:01:40.125
<00:01:37.765>to <00:01:38.045>briefly <00:01:38.325>review <00:01:38.605>Apache <00:01:38.885>Kafka <00:01:39.165>architecture.

27
00:01:40.345 --> 00:01:41.845
<00:01:40.345>So <00:01:40.537>we <00:01:40.730>are <00:01:40.923>all <00:01:41.116>on <00:01:41.309>the <00:01:41.502>same <00:01:41.695>page.

28
00:01:42.985 --> 00:01:45.085
<00:01:42.985>Uh, <00:01:43.265>discuss <00:01:43.545>latency <00:01:43.825>and <00:01:44.105>throughput <00:01:44.385>goals

29
00:01:45.305 --> 00:01:47.485
<00:01:45.305>and <00:01:45.505>how <00:01:45.705>we <00:01:45.905>can <00:01:46.105>achieve <00:01:46.305>them <00:01:46.505>with <00:01:46.705>Kafka.

30
00:01:48.505 --> 00:01:52.485
<00:01:48.505>And <00:01:48.705>then <00:01:48.905>examine <00:01:49.104>benchmarking <00:01:49.304>techniques <00:01:49.505>and <00:01:49.705>popular <00:01:49.905>tools.

31
00:01:56.845 --> 00:02:01.775
<00:01:56.845>So, <00:01:57.000>Apache <00:01:57.156>Kafka <00:01:57.311>fundamentals, <00:01:57.467>uh, <00:01:57.622>well <00:01:57.778>at <00:01:57.933>the <00:01:58.089>very <00:01:58.245>heart

32
00:02:01.775 --> 00:02:03.695
<00:02:01.775>of <00:02:02.125>Apache <00:02:02.475>Kafka <00:02:02.825>is <00:02:03.175>events.

33
00:02:04.435 --> 00:02:07.495
<00:02:04.435>Uh, <00:02:04.610>they <00:02:04.785>can <00:02:04.960>be <00:02:05.135>human <00:02:05.310>generated, <00:02:05.485>like <00:02:05.660>online <00:02:05.835>orders

34
00:02:07.675 --> 00:02:11.455
<00:02:07.675>or <00:02:07.875>clicks <00:02:08.074>or <00:02:08.275>machine <00:02:08.475>generated <00:02:08.674>coming <00:02:08.875>from <00:02:09.074>it,

35
00:02:11.455 --> 00:02:12.935
<00:02:11.455>devices <00:02:12.121>or <00:02:12.787>applications.

36
00:02:15.155 --> 00:02:18.005
<00:02:15.155>Each <00:02:15.330>event, <00:02:15.505>uh, <00:02:15.680>basically <00:02:15.855>is <00:02:16.030>a <00:02:16.205>key <00:02:16.380>value <00:02:16.555>pair.

37
00:02:19.425 --> 00:02:23.775
<00:02:19.425>Uh, <00:02:19.580>though <00:02:19.736>you <00:02:19.891>can <00:02:20.047>have <00:02:20.202>events <00:02:20.358>without <00:02:20.513>a <00:02:20.669>key, <00:02:20.825>uh,

38
00:02:23.995 --> 00:02:27.935
<00:02:23.995>the <00:02:24.135>lock <00:02:24.275>is <00:02:24.415>the <00:02:24.555>most <00:02:24.695>crucial <00:02:24.835>building <00:02:24.975>block <00:02:25.115>of <00:02:25.255>Apache <00:02:25.395>Kafka.

39
00:02:28.715 --> 00:02:32.825
<00:02:28.715>And <00:02:28.870>when <00:02:29.026>I <00:02:29.181>say <00:02:29.337>lock, <00:02:29.492>I <00:02:29.648>don't <00:02:29.803>mean <00:02:29.959>application <00:02:30.115>lock,

40
00:02:32.855 --> 00:02:34.465
<00:02:32.855>like <00:02:33.135>lock <00:02:33.415>for <00:02:33.695>J <00:02:33.975>in <00:02:34.255>Java.

41
00:02:35.335 --> 00:02:37.625
<00:02:35.335>Here <00:02:35.510>the <00:02:35.685>log <00:02:35.860>is <00:02:36.035>when <00:02:36.210>we <00:02:36.385>take <00:02:36.560>our <00:02:36.735>events

42
00:02:37.805 --> 00:02:42.185
<00:02:37.805>and <00:02:37.945>append <00:02:38.085>them <00:02:38.225>to <00:02:38.365>the <00:02:38.505>end <00:02:38.645>of <00:02:38.785>the <00:02:38.925>file <00:02:39.065>over <00:02:39.205>time,

43
00:02:43.435 --> 00:02:46.705
<00:02:43.435>which <00:02:43.635>guarantees <00:02:43.835>that, <00:02:44.035>uh, <00:02:44.235>events <00:02:44.435>will <00:02:44.635>never <00:02:44.835>change.

44
00:02:48.725 --> 00:02:51.385
<00:02:48.725>Uh, <00:02:48.924>the <00:02:49.125>log <00:02:49.325>is <00:02:49.525>immutable <00:02:49.725>and <00:02:49.924>depend <00:02:50.125>only,

45
00:02:54.405 --> 00:02:57.305
<00:02:54.405>and <00:02:54.638>Kafka <00:02:54.871>organizes, <00:02:55.105>uh, <00:02:55.338>data <00:02:55.571>into <00:02:55.805>topics.

46
00:02:59.305 --> 00:03:03.245
<00:02:59.305>And, <00:02:59.445>uh, <00:02:59.585>topic <00:02:59.725>is <00:02:59.865>a <00:03:00.005>concept <00:03:00.145>that <00:03:00.285>provides <00:03:00.425>facilities <00:03:00.565>to <00:03:00.705>work

47
00:03:03.245 --> 00:03:06.325
<00:03:03.245>with <00:03:03.445>an <00:03:03.645>immutable, <00:03:03.845>append <00:03:04.045>only <00:03:04.245>and <00:03:04.445>distributed <00:03:04.645>log.

48
00:03:07.705 --> 00:03:09.885
<00:03:07.705>Uh, <00:03:07.938>but <00:03:08.171>topic <00:03:08.405>is <00:03:08.638>a <00:03:08.871>logical <00:03:09.105>concept.

49
00:03:10.545 --> 00:03:13.205
<00:03:10.545>Topics <00:03:10.894>are <00:03:11.244>organized <00:03:11.595>into <00:03:11.945>partitions

50
00:03:13.225 --> 00:03:15.085
<00:03:13.225>to <00:03:13.458>make <00:03:13.691>the <00:03:13.924>log <00:03:14.158>distributed <00:03:14.391>and <00:03:14.625>scalable

51
00:03:16.545 --> 00:03:18.965
<00:03:16.545>and <00:03:17.011>messages <00:03:17.478>are <00:03:17.945>guaranteed

52
00:03:18.965 --> 00:03:21.645
<00:03:18.965>to <00:03:19.120>be <00:03:19.276>strictly <00:03:19.431>ordered <00:03:19.587>within <00:03:19.742>a <00:03:19.898>partition, <00:03:20.053>within <00:03:20.209>a <00:03:20.365>topic.

53
00:03:25.195 --> 00:03:29.015
<00:03:25.195>To <00:03:25.322>make <00:03:25.449>your <00:03:25.576>data <00:03:25.704>durable, <00:03:25.831>uh, <00:03:25.958>you <00:03:26.085>can <00:03:26.213>have <00:03:26.340>as <00:03:26.467>many <00:03:26.595>replicas

54
00:03:29.155 --> 00:03:30.935
<00:03:29.155>as <00:03:29.435>you <00:03:29.715>want, <00:03:29.995>but <00:03:30.275>typically <00:03:30.555>three

55
00:03:30.935 --> 00:03:35.195
<00:03:30.935>or <00:03:31.062>four, <00:03:31.189>uh, <00:03:31.316>one <00:03:31.444>of <00:03:31.571>those <00:03:31.698>replicas <00:03:31.825>is <00:03:31.953>assigned <00:03:32.080>as <00:03:32.207>a <00:03:32.335>leader

56
00:03:35.375 --> 00:03:37.155
<00:03:35.375>and <00:03:35.841>others <00:03:36.308>are <00:03:36.775>follows.

57
00:03:38.705 --> 00:03:41.765
<00:03:38.704>And <00:03:38.904>if <00:03:39.105>the, <00:03:39.304>if <00:03:39.505>the <00:03:39.704>leader <00:03:39.904>fails, <00:03:40.105>one

58
00:03:41.765 --> 00:03:43.245
<00:03:41.765>of <00:03:42.097>its <00:03:42.430>replicas <00:03:42.764>is <00:03:43.097>promoted.

59
00:03:44.855 --> 00:03:47.395
<00:03:44.855>Uh, <00:03:45.030>this <00:03:45.204>is <00:03:45.380>basically <00:03:45.554>how <00:03:45.730>we <00:03:45.905>get <00:03:46.079>our <00:03:46.255>durability

60
00:03:47.495 --> 00:03:49.395
<00:03:47.495>and <00:03:47.845>availability <00:03:48.195>in <00:03:48.545>Apache <00:03:48.895>Kafka.

61
00:03:53.055 --> 00:03:56.035
<00:03:53.055>So <00:03:53.195>how <00:03:53.335>do <00:03:53.475>you <00:03:53.615>get <00:03:53.755>events <00:03:53.895>into <00:03:54.035>the <00:03:54.175>distributed <00:03:54.315>commit <00:03:54.455>log

62
00:03:56.995 --> 00:03:58.775
<00:03:56.995>to <00:03:57.275>get <00:03:57.555>data <00:03:57.835>into <00:03:58.115>a <00:03:58.395>topic?

63
00:03:58.875 --> 00:04:00.575
<00:03:58.875>In <00:03:59.155>Kafka, <00:03:59.435>we <00:03:59.715>use <00:03:59.995>a <00:04:00.275>producer.

64
00:04:01.475 --> 00:04:03.855
<00:04:01.475>Uh, <00:04:01.614>the <00:04:01.755>producer <00:04:01.894>writes <00:04:02.035>data <00:04:02.174>to <00:04:02.315>the <00:04:02.454>end <00:04:02.595>of <00:04:02.734>the <00:04:02.875>log

65
00:04:04.435 --> 00:04:06.855
<00:04:04.435>and <00:04:04.715>the <00:04:04.995>producer <00:04:05.275>decides <00:04:05.555>which <00:04:05.835>partition

66
00:04:06.995 --> 00:04:08.055
<00:04:06.995>are <00:04:07.233>we <00:04:07.472>going <00:04:07.710>to <00:04:07.949>write.

67
00:04:09.435 --> 00:04:11.775
<00:04:09.435>So <00:04:09.635>producers <00:04:09.835>are <00:04:10.035>always <00:04:10.235>write <00:04:10.435>into <00:04:10.635>the <00:04:10.835>leader

68
00:04:12.355 --> 00:04:15.335
<00:04:12.355>and <00:04:12.635>messages <00:04:12.915>without <00:04:13.195>a <00:04:13.475>key, <00:04:13.755>uh,

69
00:04:15.695 --> 00:04:17.215
<00:04:15.695>produced <00:04:16.036>in <00:04:16.378>round <00:04:16.721>robin <00:04:17.063>fashion.

70
00:04:18.725 --> 00:04:22.785
<00:04:18.725>Uh, <00:04:18.865>but <00:04:19.005>messages <00:04:19.145>produced <00:04:19.285>with <00:04:19.425>a <00:04:19.565>key, <00:04:19.705>uh, <00:04:19.845>will <00:04:19.985>be <00:04:20.125>assigned

71
00:04:22.965 --> 00:04:24.385
<00:04:22.965>to <00:04:23.604>particular <00:04:24.243>partitions

72
00:04:26.405 --> 00:04:27.985
<00:04:26.405>and, <00:04:27.104>uh, <00:04:27.804>messages

73
00:04:28.005 --> 00:04:31.105
<00:04:28.005>for <00:04:28.132>a <00:04:28.259>given <00:04:28.386>key <00:04:28.514>will <00:04:28.641>always <00:04:28.768>end <00:04:28.895>up <00:04:29.023>on <00:04:29.150>the <00:04:29.277>same <00:04:29.405>partition.

74
00:04:39.385 --> 00:04:42.405
<00:04:39.385>So <00:04:39.512>to <00:04:39.639>get <00:04:39.766>data <00:04:39.894>out <00:04:40.021>of <00:04:40.148>the <00:04:40.275>topic, <00:04:40.403>uh, <00:04:40.530>we <00:04:40.657>are <00:04:40.784>going

75
00:04:42.405 --> 00:04:44.125
<00:04:42.405>to <00:04:42.871>use <00:04:43.338>consumer <00:04:43.804>API.

76
00:04:44.425 --> 00:04:46.645
<00:04:44.425>And <00:04:44.625>when <00:04:44.825>we <00:04:45.025>consume <00:04:45.225>data <00:04:45.425>from <00:04:45.625>a <00:04:45.825>topic,

77
00:04:46.825 --> 00:04:48.525
<00:04:46.825>we <00:04:47.104>do <00:04:47.385>it <00:04:47.664>in <00:04:47.945>sequential <00:04:48.224>order.

78
00:04:49.905 --> 00:04:51.805
<00:04:49.905>So <00:04:50.255>consumers <00:04:50.604>regularly <00:04:50.955>commit <00:04:51.304>offsets

79
00:04:51.805 --> 00:04:52.885
<00:04:51.805>to <00:04:52.129>track <00:04:52.453>their <00:04:52.777>progress,

80
00:04:54.185 --> 00:04:56.605
<00:04:54.185>and <00:04:54.465>there <00:04:54.745>can <00:04:55.025>be <00:04:55.305>multiple <00:04:55.585>consumers.

81
00:04:56.785 --> 00:05:00.165
<00:04:56.784>And <00:04:56.940>Kafka <00:04:57.096>will <00:04:57.251>track <00:04:57.407>how <00:04:57.562>far <00:04:57.718>each <00:04:57.873>consumer <00:04:58.029>has <00:04:58.184>got.

82
00:05:01.985 --> 00:05:04.285
<00:05:01.985>If <00:05:02.160>you <00:05:02.335>have <00:05:02.510>a <00:05:02.685>single <00:05:02.860>instance <00:05:03.035>of <00:05:03.210>a <00:05:03.385>consumer,

83
00:05:04.815 --> 00:05:07.085
<00:05:04.815>it'll <00:05:05.094>read <00:05:05.375>data <00:05:05.655>from <00:05:05.935>every <00:05:06.215>partition.

84
00:05:08.785 --> 00:05:10.645
<00:05:08.785>Uh, <00:05:09.065>but <00:05:09.345>we <00:05:09.625>can <00:05:09.905>scale, <00:05:10.185>uh,

85
00:05:11.045 --> 00:05:13.085
<00:05:11.045>consumers <00:05:11.325>horizontally <00:05:11.605>when <00:05:11.885>we <00:05:12.165>assemble <00:05:12.445>into,

86
00:05:13.085 --> 00:05:14.125
<00:05:13.085>into <00:05:13.553>consumer <00:05:14.021>groups.

87
00:05:15.625 --> 00:05:19.245
<00:05:15.625>And, <00:05:15.752>uh, <00:05:15.879>so <00:05:16.006>each <00:05:16.134>of <00:05:16.261>those <00:05:16.388>consumers <00:05:16.515>will <00:05:16.643>get <00:05:16.770>data <00:05:16.897>from <00:05:17.025>one

88
00:05:19.245 --> 00:05:24.005
<00:05:19.245>or <00:05:19.525>more <00:05:19.805>partitions <00:05:20.085>and <00:05:20.365>Kafka <00:05:20.645>keeps

89
00:05:24.015 --> 00:05:26.965
<00:05:24.015>track <00:05:24.155>of <00:05:24.294>how <00:05:24.435>many <00:05:24.575>consumers <00:05:24.715>you <00:05:24.854>have <00:05:24.995>in <00:05:25.135>the <00:05:25.275>consumer <00:05:25.414>group

90
00:05:27.185 --> 00:05:29.725
<00:05:27.185>and <00:05:27.465>rebalance <00:05:27.745>them <00:05:28.025>if <00:05:28.305>you <00:05:28.585>add

91
00:05:29.825 --> 00:05:31.605
<00:05:29.825>or <00:05:30.104>remove <00:05:30.385>one <00:05:30.664>of <00:05:30.945>the <00:05:31.224>consumers.

92
00:05:42.085 --> 00:05:44.425
<00:05:42.085>Uh, <00:05:42.364>so <00:05:42.645>all <00:05:42.924>systems <00:05:43.205>are <00:05:43.484>different.

93
00:05:44.445 --> 00:05:46.185
<00:05:44.445>And <00:05:44.678>to <00:05:44.911>tuner <00:05:45.145>Apache <00:05:45.378>Kafka, <00:05:45.611>you <00:05:45.844>need

94
00:05:46.185 --> 00:05:47.625
<00:05:46.185>to <00:05:46.617>understand <00:05:47.049>your <00:05:47.481>workloads.

95
00:05:48.885 --> 00:05:51.825
<00:05:48.885>How <00:05:49.118>does <00:05:49.351>one <00:05:49.585>get <00:05:49.818>the <00:05:50.051>most <00:05:50.284>throughput

96
00:05:51.825 --> 00:05:54.185
<00:05:51.825>and <00:05:52.058>lowest <00:05:52.291>latency <00:05:52.525>from <00:05:52.758>Apache <00:05:52.991>Kafka <00:05:53.224>cluster

97
00:05:54.565 --> 00:05:57.185
<00:05:54.565>or <00:05:54.705>how <00:05:54.844>much <00:05:54.985>throughput <00:05:55.125>you <00:05:55.265>can <00:05:55.405>get <00:05:55.545>from <00:05:55.685>a <00:05:55.825>single <00:05:55.965>partition?

98
00:05:57.715 --> 00:06:02.605
<00:05:57.715>Understanding <00:05:57.948>your <00:05:58.181>systems, <00:05:58.415>uh, <00:05:58.648>helps <00:05:58.881>in <00:05:59.115>benchmarking

99
00:06:02.605 --> 00:06:06.405
<00:06:02.605>and <00:06:02.780>tune <00:06:02.955>Apache <00:06:03.130>Kafka <00:06:03.305>as <00:06:03.480>you <00:06:03.655>can <00:06:03.830>shape <00:06:04.005>data

100
00:06:07.865 --> 00:06:10.365
<00:06:07.865>and <00:06:08.145>workloads <00:06:08.425>to <00:06:08.705>your <00:06:08.985>actual <00:06:09.265>requirements.

101
00:06:13.395 --> 00:06:15.455
<00:06:13.395>Uh, <00:06:13.674>then <00:06:13.955>it <00:06:14.234>would <00:06:14.515>help, <00:06:14.794>uh,

102
00:06:15.475 --> 00:06:18.295
<00:06:15.475>if <00:06:15.630>you <00:06:15.786>decide <00:06:15.941>which <00:06:16.097>service <00:06:16.252>goals <00:06:16.408>you <00:06:16.563>want <00:06:16.719>to <00:06:16.875>optimize.

103
00:06:19.275 --> 00:06:23.375
<00:06:19.275>Uh, <00:06:19.414>so <00:06:19.554>here <00:06:19.695>in <00:06:19.835>this <00:06:19.974>talk <00:06:20.114>I <00:06:20.255>will <00:06:20.395>consider <00:06:20.534>two <00:06:20.674>goals

104
00:06:23.525 --> 00:06:25.935
<00:06:23.525>that <00:06:23.758>often <00:06:23.991>involve <00:06:24.224>trade-offs <00:06:24.458>with <00:06:24.691>one <00:06:24.924>another,

105
00:06:26.625 --> 00:06:28.015
<00:06:26.625>throughput <00:06:27.250>and <00:06:27.876>latency.

106
00:06:29.715 --> 00:06:32.415
<00:06:29.715>And <00:06:29.854>to <00:06:29.994>figure <00:06:30.135>out <00:06:30.275>which <00:06:30.414>goals <00:06:30.554>you <00:06:30.695>want <00:06:30.835>to <00:06:30.974>optimize, <00:06:31.114>uh,

107
00:06:32.475 --> 00:06:35.295
<00:06:32.475>recall <00:06:32.615>the <00:06:32.755>use <00:06:32.895>cases <00:06:33.035>your <00:06:33.175>cluster <00:06:33.315>is <00:06:33.455>going <00:06:33.595>to <00:06:33.735>serve, <00:06:33.875>serve,

108
00:06:37.275 --> 00:06:40.505
<00:06:37.275>there <00:06:37.450>are <00:06:37.625>c <00:06:37.799>configuration <00:06:37.974>parameters <00:06:38.150>that <00:06:38.325>you <00:06:38.500>should <00:06:38.674>change

109
00:06:40.565 --> 00:06:42.985
<00:06:40.565>to <00:06:40.844>optimize <00:06:41.125>for <00:06:41.405>those <00:06:41.685>service <00:06:41.965>goals.

110
00:06:45.075 --> 00:06:47.695
<00:06:45.075>You <00:06:45.230>need <00:06:45.386>to <00:06:45.541>understand <00:06:45.697>what <00:06:45.852>your <00:06:46.008>users <00:06:46.163>expect <00:06:46.319>from <00:06:46.474>the

111
00:06:47.695 --> 00:06:49.375
<00:06:47.695>system <00:06:48.395>and, <00:06:49.094>uh,

112
00:06:49.475 --> 00:06:52.055
<00:06:49.475>to <00:06:49.650>ensure <00:06:49.825>you're <00:06:50.000>optimizing <00:06:50.175>c <00:06:50.350>to <00:06:50.525>meet <00:06:50.700>their <00:06:50.875>needs.

113
00:06:53.985 --> 00:06:56.245
<00:06:53.985>So <00:06:54.160>do <00:06:54.335>you <00:06:54.510>want <00:06:54.685>to <00:06:54.860>optimize <00:06:55.035>for <00:06:55.210>high <00:06:55.385>throughput,

114
00:06:56.245 --> 00:07:00.005
<00:06:56.245>which <00:06:56.385>is <00:06:56.525>the <00:06:56.665>rate <00:06:56.805>the <00:06:56.945>data <00:06:57.085>moved <00:06:57.225>from <00:06:57.365>producers <00:06:57.505>to <00:06:57.645>brokers

115
00:07:00.025 --> 00:07:01.325
<00:07:00.025>or <00:07:00.414>brokers <00:07:00.805>to <00:07:01.195>consumers?

116
00:07:02.275 --> 00:07:05.405
<00:07:02.275>Because <00:07:02.474>some <00:07:02.674>use <00:07:02.875>cases <00:07:03.075>can <00:07:03.275>have <00:07:03.474>like <00:07:03.674>millions

117
00:07:05.405 --> 00:07:06.885
<00:07:05.405>of <00:07:05.671>rights <00:07:05.937>or <00:07:06.204>reads <00:07:06.470>per <00:07:06.736>second.

118
00:07:08.185 --> 00:07:10.485
<00:07:08.185>Or <00:07:08.360>do <00:07:08.535>you <00:07:08.710>want <00:07:08.885>to <00:07:09.060>optimize <00:07:09.235>for <00:07:09.410>low <00:07:09.585>latency

119
00:07:10.825 --> 00:07:12.605
<00:07:10.825>or <00:07:11.175>which <00:07:11.525>is <00:07:11.875>the <00:07:12.224>elapsed?

120
00:07:12.865 --> 00:07:17.445
<00:07:12.865>Uh, <00:07:13.040>time <00:07:13.215>moving <00:07:13.390>messages <00:07:13.565>end <00:07:13.740>to <00:07:13.915>end <00:07:14.090>from <00:07:14.265>producers

121
00:07:17.465 --> 00:07:19.565
<00:07:17.465>to <00:07:17.931>brokers <00:07:18.398>to <00:07:18.864>consumers,

122
00:07:22.615 --> 00:07:25.525
<00:07:22.615>there <00:07:22.815>are <00:07:23.015>hundreds <00:07:23.215>of <00:07:23.415>different <00:07:23.615>configuration <00:07:23.815>parameters <00:07:24.015>in

123
00:07:25.525 --> 00:07:27.645
<00:07:25.525>Apache <00:07:25.804>Kafka, <00:07:26.085>and <00:07:26.364>we <00:07:26.645>are <00:07:26.924>going

124
00:07:27.645 --> 00:07:29.365
<00:07:27.645>to <00:07:27.844>discuss <00:07:28.044>just <00:07:28.245>a <00:07:28.445>small <00:07:28.645>subset <00:07:28.844>of <00:07:29.044>them.

125
00:07:30.475 --> 00:07:33.975
<00:07:30.475>Uh, <00:07:30.755>remember, <00:07:31.035>benchmarking <00:07:31.315>is <00:07:31.595>always <00:07:31.875>important

126
00:07:33.995 --> 00:07:36.535
<00:07:33.995>to <00:07:34.195>validate <00:07:34.395>the <00:07:34.595>settings <00:07:34.795>for <00:07:34.995>your <00:07:35.195>specific <00:07:35.395>deployment.

127
00:07:40.425 --> 00:07:42.085
<00:07:40.425>So <00:07:41.125>end-to-end <00:07:41.825>latency.

128
00:07:42.505 --> 00:07:44.805
<00:07:42.505>Uh, <00:07:42.680>when <00:07:42.855>we <00:07:43.030>talk <00:07:43.205>about <00:07:43.380>latency, <00:07:43.555>we <00:07:43.730>talk <00:07:43.905>about

129
00:07:45.375 --> 00:07:48.245
<00:07:45.375>end-to-end <00:07:45.575>latency <00:07:45.775>usually, <00:07:45.975>but <00:07:46.175>what <00:07:46.375>does <00:07:46.575>that <00:07:46.775>mean?

130
00:07:48.505 --> 00:07:52.165
<00:07:48.505>Uh, <00:07:48.680>well <00:07:48.855>benchmarking <00:07:49.030>Apache <00:07:49.205>Kafka, <00:07:49.380>we <00:07:49.555>should <00:07:49.730>measure <00:07:49.905>times,

131
00:07:52.625 --> 00:07:56.605
<00:07:52.625>uh, <00:07:52.825>between <00:07:53.025>measure <00:07:53.225>send <00:07:53.425>using <00:07:53.625>producer <00:07:53.825>send <00:07:54.025>method

132
00:07:57.175 --> 00:08:00.995
<00:07:57.175>until <00:07:57.875>uh, <00:07:58.575>uh,

133
00:08:02.035 --> 00:08:05.035
<00:08:02.035>consumer <00:08:02.315>poll, <00:08:02.595>uh, <00:08:02.875>returns <00:08:03.155>that <00:08:03.435>record.

134
00:08:06.935 --> 00:08:11.885
<00:08:06.935>Uh, <00:08:07.215>but <00:08:07.495>in <00:08:07.775>order <00:08:08.055>to <00:08:08.335>optimize

135
00:08:12.465 --> 00:08:16.005
<00:08:12.465>Apache <00:08:12.640>Kafka <00:08:12.815>and <00:08:12.989>to <00:08:13.164>measure <00:08:13.340>the <00:08:13.515>latency, <00:08:13.690>we <00:08:13.864>need

136
00:08:16.005 --> 00:08:17.565
<00:08:16.005>to <00:08:16.284>understand <00:08:16.565>what <00:08:16.844>happens <00:08:17.125>in <00:08:17.405>between.

137
00:08:19.105 --> 00:08:22.525
<00:08:19.105>Uh, <00:08:19.385>so <00:08:19.665>produce <00:08:19.945>time <00:08:20.225>is, <00:08:20.505>uh,

138
00:08:23.315 --> 00:08:25.085
<00:08:23.315>time <00:08:23.665>spent <00:08:24.015>processing <00:08:24.365>the <00:08:24.715>record

139
00:08:25.225 --> 00:08:28.365
<00:08:25.225>and <00:08:25.380>batching <00:08:25.536>it, <00:08:25.691>uh, <00:08:25.847>with <00:08:26.002>other <00:08:26.158>events <00:08:26.313>in <00:08:26.469>Kafka <00:08:26.625>producer.

140
00:08:28.985 --> 00:08:30.205
<00:08:28.985>Uh, <00:08:29.351>then <00:08:29.717>publish <00:08:30.082>time,

141
00:08:30.495 --> 00:08:33.725
<00:08:30.495>which <00:08:30.650>is <00:08:30.806>sending <00:08:30.961>the <00:08:31.117>record <00:08:31.272>from <00:08:31.428>Kafka <00:08:31.583>producer <00:08:31.739>to <00:08:31.895>broker.

142
00:08:34.385 --> 00:08:38.885
<00:08:34.385>And <00:08:34.540>depending <00:08:34.696>into <00:08:34.851>the <00:08:35.007>leader <00:08:35.162>log <00:08:35.318>commit <00:08:35.473>time <00:08:35.629>is <00:08:35.784>uh,

143
00:08:38.885 --> 00:08:41.205
<00:08:38.885>replicating <00:08:39.164>the <00:08:39.444>record <00:08:39.725>to <00:08:40.005>follow <00:08:40.284>replicas

144
00:08:41.385 --> 00:08:46.265
<00:08:41.385>for <00:08:41.540>full <00:08:41.696>tolerance, <00:08:41.851>uh, <00:08:42.007>which <00:08:42.162>is <00:08:42.318>basically, <00:08:42.473>uh, <00:08:42.629>the <00:08:42.784>time

145
00:08:46.265 --> 00:08:47.985
<00:08:46.265>of <00:08:46.498>the <00:08:46.731>slowest <00:08:46.965>replica <00:08:47.198>to <00:08:47.431>catch <00:08:47.665>up.

146
00:08:49.605 --> 00:08:51.865
<00:08:49.605>Uh, <00:08:49.838>then <00:08:50.071>catch <00:08:50.305>up <00:08:50.538>time <00:08:50.771>catching <00:08:51.005>up

147
00:08:51.865 --> 00:08:53.705
<00:08:51.865>to <00:08:52.098>the <00:08:52.331>records <00:08:52.565>offset <00:08:52.798>in <00:08:53.031>the <00:08:53.265>log.

148
00:08:55.175 --> 00:08:56.875
<00:08:55.175>And, <00:08:55.641>uh, <00:08:56.108>fetch <00:08:56.574>time

149
00:08:57.935 --> 00:09:00.235
<00:08:57.935>is <00:08:58.135>actually <00:08:58.334>fetching <00:08:58.535>the <00:08:58.734>record <00:08:58.935>from <00:08:59.135>the <00:08:59.334>broker.

150
00:09:05.185 --> 00:09:08.805
<00:09:05.185>So <00:09:05.651>optimizing <00:09:06.118>latency, <00:09:06.584>um,

151
00:09:08.875 --> 00:09:12.365
<00:09:08.875>each <00:09:09.050>Kaf <00:09:09.225>Kafka <00:09:09.400>topic <00:09:09.575>contains <00:09:09.750>one <00:09:09.925>or <00:09:10.100>more <00:09:10.275>partitions.

152
00:09:13.025 --> 00:09:17.485
<00:09:13.025>And, <00:09:13.180>uh, <00:09:13.336>when <00:09:13.491>Kafka <00:09:13.647>producer <00:09:13.802>sends <00:09:13.958>a <00:09:14.113>record <00:09:14.269>to <00:09:14.425>topic,

153
00:09:17.825 --> 00:09:20.165
<00:09:17.825>it <00:09:18.000>needs <00:09:18.175>to <00:09:18.350>decide <00:09:18.525>which <00:09:18.700>partition <00:09:18.875>to <00:09:19.050>send <00:09:19.225>it.

154
00:09:22.185 --> 00:09:25.005
<00:09:22.185>If <00:09:22.340>we <00:09:22.496>send <00:09:22.651>several <00:09:22.807>records <00:09:22.962>to <00:09:23.118>the <00:09:23.273>same <00:09:23.429>partition <00:09:23.584>at

155
00:09:25.005 --> 00:09:28.125
<00:09:25.005>around <00:09:25.145>the <00:09:25.285>same <00:09:25.425>time, <00:09:25.564>they <00:09:25.705>can <00:09:25.845>be <00:09:25.985>sent <00:09:26.125>as <00:09:26.265>a <00:09:26.405>batch

156
00:09:28.905 --> 00:09:31.685
<00:09:28.905>and <00:09:29.138>processing <00:09:29.371>each <00:09:29.605>batch <00:09:29.838>requires <00:09:30.071>some <00:09:30.305>overhead.

157
00:09:32.755 --> 00:09:35.765
<00:09:32.755>With <00:09:32.930>each <00:09:33.105>of <00:09:33.280>the <00:09:33.455>records <00:09:33.630>inside <00:09:33.805>the <00:09:33.980>batch <00:09:34.155>contributing

158
00:09:35.765 --> 00:09:38.945
<00:09:35.765>to <00:09:36.231>the <00:09:36.698>cost, <00:09:37.165>uh,

159
00:09:39.075 --> 00:09:43.025
<00:09:39.075>batch <00:09:39.230>is <00:09:39.386>completed <00:09:39.541>either <00:09:39.697>when <00:09:39.852>it <00:09:40.008>reaches <00:09:40.163>a <00:09:40.319>specific <00:09:40.475>size,

160
00:09:44.765 --> 00:09:48.305
<00:09:44.765>it <00:09:44.965>is <00:09:45.165>a <00:09:45.365>batch <00:09:45.564>size <00:09:45.765>configuration <00:09:45.965>parameter, <00:09:46.165>or

161
00:09:48.305 --> 00:09:49.945
<00:09:48.305>after <00:09:48.771>some <00:09:49.238>time, <00:09:49.704>uh,

162
00:09:50.045 --> 00:09:52.345
<00:09:50.045>it <00:09:50.278>is <00:09:50.511>a <00:09:50.745>linger <00:09:50.978>MS <00:09:51.211>configuration <00:09:51.444>parameter.

163
00:09:53.925 --> 00:09:58.825
<00:09:53.925>Uh, <00:09:54.064>by <00:09:54.204>default, <00:09:54.344>uh, <00:09:54.484>the <00:09:54.625>producer <00:09:54.765>is <00:09:54.905>tuned <00:09:55.045>for <00:09:55.185>low <00:09:55.324>for,

164
00:09:58.875 --> 00:10:01.585
<00:09:58.875>sorry, <00:09:59.075>the <00:09:59.275>producer <00:09:59.475>is <00:09:59.675>tuned <00:09:59.875>for <00:10:00.075>low <00:10:00.275>latency

165
00:10:02.485 --> 00:10:05.825
<00:10:02.485>and <00:10:02.640>the <00:10:02.796>configuration <00:10:02.951>parameter <00:10:03.107>link <00:10:03.262>ms <00:10:03.418>is <00:10:03.573>set <00:10:03.729>to <00:10:03.885>zero,

166
00:10:07.395 --> 00:10:10.985
<00:10:07.395>which <00:10:07.569>basically <00:10:07.745>means <00:10:07.920>the <00:10:08.095>producer <00:10:08.270>will <00:10:08.444>send <00:10:08.620>as <00:10:08.795>soon

167
00:10:11.205 --> 00:10:13.345
<00:10:11.205>as <00:10:11.485>it <00:10:11.765>has <00:10:12.045>data <00:10:12.325>to <00:10:12.605>send.

168
00:10:16.555 --> 00:10:20.295
<00:10:16.555>Also, <00:10:16.755>consider <00:10:16.954>whether <00:10:17.155>you <00:10:17.354>need <00:10:17.555>to <00:10:17.755>enable <00:10:17.954>compression.

169
00:10:20.875 --> 00:10:24.655
<00:10:20.875>So <00:10:21.075>enabling <00:10:21.275>compression <00:10:21.475>ally <00:10:21.675>requires <00:10:21.875>more <00:10:22.075>CPU <00:10:22.275>cycles

170
00:10:24.655 --> 00:10:26.975
<00:10:24.655>to <00:10:24.855>do <00:10:25.055>the <00:10:25.255>compression, <00:10:25.454>but <00:10:25.655>it <00:10:25.855>reduces <00:10:26.055>network

171
00:10:27.445 --> 00:10:28.935
<00:10:27.445>bandwidth <00:10:28.786>utilization.

172
00:10:30.795 --> 00:10:35.255
<00:10:30.795>Uh, <00:10:30.969>the <00:10:31.145>record <00:10:31.319>is <00:10:31.495>acknowledged <00:10:31.670>based <00:10:31.844>on <00:10:32.020>X <00:10:32.194>configuration,

173
00:10:35.825 --> 00:10:38.495
<00:10:35.825>which <00:10:36.105>controls <00:10:36.385>the <00:10:36.665>durability <00:10:36.945>of <00:10:37.225>records.

174
00:10:40.045 --> 00:10:43.585
<00:10:40.045>Uh, <00:10:40.185>there <00:10:40.324>are <00:10:40.464>three <00:10:40.604>main, <00:10:40.745>uh, <00:10:40.885>yeah, <00:10:41.025>there <00:10:41.165>are <00:10:41.305>three <00:10:41.444>options.

175
00:10:43.775 --> 00:10:47.665
<00:10:43.775>Basically, <00:10:44.125>uh, <00:10:44.475>x <00:10:44.824>equals <00:10:45.175>zero,

176
00:10:48.395 --> 00:10:53.105
<00:10:48.395>which <00:10:48.628>is, <00:10:48.861>uh, <00:10:49.095>like <00:10:49.328>reply <00:10:49.561>immediately, <00:10:49.795>uh,

177
00:10:53.185 --> 00:10:55.185
<00:10:53.185>without <00:10:53.385>waiting <00:10:53.584>for <00:10:53.785>a <00:10:53.984>reply <00:10:54.185>from <00:10:54.385>a <00:10:54.584>broker.

178
00:10:57.285 --> 00:11:01.985
<00:10:57.285>And <00:10:57.564>then <00:10:57.844>X <00:10:58.125>equal <00:10:58.405>one, <00:10:58.685>uh,

179
00:11:02.055 --> 00:11:03.145
<00:11:02.055>acknowledge <00:11:02.545>the <00:11:03.036>message

180
00:11:03.155 --> 00:11:04.985
<00:11:03.155>after <00:11:03.388>appending <00:11:03.621>the <00:11:03.855>message <00:11:04.088>to <00:11:04.321>the <00:11:04.555>leader,

181
00:11:05.485 --> 00:11:08.185
<00:11:05.485>and <00:11:05.835>X <00:11:06.185>equal <00:11:06.535>all, <00:11:06.885>uh,

182
00:11:08.765 --> 00:11:11.985
<00:11:08.765>to <00:11:08.965>acknowledge <00:11:09.165>the <00:11:09.365>message <00:11:09.564>only <00:11:09.765>when <00:11:09.965>old <00:11:10.165>followers

183
00:11:13.025 --> 00:11:16.405
<00:11:13.025>and, <00:11:13.258>uh, <00:11:13.491>the <00:11:13.725>leader <00:11:13.958>acknowledge <00:11:14.191>the <00:11:14.425>message.

184
00:11:19.355 --> 00:11:23.395
<00:11:19.355>Um, <00:11:19.555>it <00:11:19.755>is <00:11:19.955>a <00:11:20.155>common <00:11:20.355>belief <00:11:20.555>that, <00:11:20.755>uh,

185
00:11:23.855 --> 00:11:28.235
<00:11:23.855>to <00:11:23.995>minimize <00:11:24.135>latency, <00:11:24.275>uh, <00:11:24.415>you <00:11:24.555>need <00:11:24.695>to <00:11:24.835>set <00:11:24.975>X <00:11:25.115>equal <00:11:25.255>one,

186
00:11:29.125 --> 00:11:31.795
<00:11:29.125>which <00:11:29.325>guarantees, <00:11:29.525>uh, <00:11:29.725>right <00:11:29.925>into <00:11:30.125>the <00:11:30.325>lead <00:11:30.525>log.

187
00:11:32.735 --> 00:11:36.435
<00:11:32.735>Uh, <00:11:32.875>it <00:11:33.015>is <00:11:33.155>true <00:11:33.295>if <00:11:33.435>you <00:11:33.575>are <00:11:33.715>interested <00:11:33.855>in <00:11:33.995>producer <00:11:34.135>latency

188
00:11:36.465 --> 00:11:38.235
<00:11:36.465>only, <00:11:37.865>uh,

189
00:11:38.535 --> 00:11:43.195
<00:11:38.535>but, <00:11:38.735>um, <00:11:38.935>wrong <00:11:39.135>if <00:11:39.334>we <00:11:39.535>consider <00:11:39.735>end-to-end <00:11:39.935>latency

190
00:11:43.625 --> 00:11:46.755
<00:11:43.625>because <00:11:43.905>consumers <00:11:44.185>can't <00:11:44.465>consume <00:11:44.745>a <00:11:45.025>message

191
00:11:46.965 --> 00:11:49.835
<00:11:46.965>until <00:11:47.198>it <00:11:47.431>is <00:11:47.665>written <00:11:47.898>to <00:11:48.131>the <00:11:48.365>follows.

192
00:11:50.495 --> 00:11:53.685
<00:11:50.495>Uh, <00:11:50.670>in <00:11:50.845>Kafka, <00:11:51.020>there <00:11:51.195>are <00:11:51.370>automatic <00:11:51.545>checks <00:11:51.720>in <00:11:51.895>place

193
00:11:57.265 --> 00:12:00.085
<00:11:57.265>to <00:11:57.498>make <00:11:57.731>sure <00:11:57.965>consumers <00:11:58.198>cannot <00:11:58.431>read <00:11:58.665>messages

194
00:12:00.085 --> 00:12:01.565
<00:12:00.085>that <00:12:00.418>haven't <00:12:00.751>been <00:12:01.084>committed <00:12:01.417>yet.

195
00:12:03.705 --> 00:12:07.925
<00:12:03.705>Uh, <00:12:03.845>so <00:12:03.985>basically <00:12:04.125>a <00:12:04.265>certain <00:12:04.405>X <00:12:04.545>to <00:12:04.685>one, <00:12:04.825>you <00:12:04.965>are <00:12:05.105>sacrificing

196
00:12:08.445 --> 00:12:10.125
<00:12:08.445>sacrificing <00:12:08.795>data <00:12:09.145>durability <00:12:09.495>for <00:12:09.845>nothing.

197
00:12:13.135 --> 00:12:17.105
<00:12:13.135>Also, <00:12:13.275>you <00:12:13.415>can <00:12:13.555>tune, <00:12:13.694>consumers <00:12:13.835>follow <00:12:13.975>a <00:12:14.115>latency <00:12:14.255>by <00:12:14.395>just <00:12:14.535>in

198
00:12:17.125 --> 00:12:20.025
<00:12:17.125>how <00:12:17.300>much <00:12:17.475>data <00:12:17.650>it <00:12:17.825>gets, <00:12:18.000>uh, <00:12:18.175>from <00:12:18.350>each <00:12:18.525>fetch.

199
00:12:21.315 --> 00:12:22.335
<00:12:21.315>Uh, <00:12:21.774>by <00:12:22.233>default,

200
00:12:22.395 --> 00:12:25.775
<00:12:22.395>the <00:12:22.569>consumer <00:12:22.745>configuration <00:12:22.920>parameter <00:12:23.095>fetch <00:12:23.270>mean <00:12:23.444>bytes <00:12:23.620>is <00:12:23.795>set

201
00:12:25.775 --> 00:12:30.655
<00:12:25.775>to <00:12:25.902>one, <00:12:26.029>and <00:12:26.156>it <00:12:26.284>means <00:12:26.411>that <00:12:26.538>fetch <00:12:26.665>requests <00:12:26.793>are <00:12:26.920>onset <00:12:27.047>as <00:12:27.175>soon

202
00:12:30.655 --> 00:12:32.815
<00:12:30.655>as <00:12:30.855>a <00:12:31.055>single <00:12:31.255>byte <00:12:31.454>of <00:12:31.655>data <00:12:31.855>is <00:12:32.055>available.

203
00:12:34.035 --> 00:12:38.695
<00:12:34.035>All <00:12:34.190>the <00:12:34.346>fetch <00:12:34.501>request <00:12:34.657>times, <00:12:34.812>times <00:12:34.968>out, <00:12:35.123>waiting <00:12:35.279>for <00:12:35.435>data

204
00:12:38.715 --> 00:12:40.535
<00:12:38.715>to <00:12:39.415>arrive, <00:12:40.115>uh,

205
00:12:40.545 --> 00:12:44.735
<00:12:40.545>there <00:12:40.719>is <00:12:40.895>a <00:12:41.069>fetch <00:12:41.245>max <00:12:41.420>wait <00:12:41.594>mass <00:12:41.770>configuration <00:12:41.944>parameter

206
00:12:46.225 --> 00:12:46.925
<00:12:46.225>to <00:12:46.540>control <00:12:46.855>that

207
00:13:00.185 --> 00:13:04.275
<00:13:00.185>partition <00:13:00.464>strategy <00:13:00.744>determines <00:13:01.025>how <00:13:01.305>batches <00:13:01.584>work.

208
00:13:05.975 --> 00:13:08.875
<00:13:05.975>If <00:13:06.150>records <00:13:06.325>are <00:13:06.500>not <00:13:06.675>sent <00:13:06.850>to <00:13:07.025>the <00:13:07.200>same <00:13:07.375>partition,

209
00:13:08.945 --> 00:13:10.995
<00:13:08.945>they <00:13:09.225>cannot <00:13:09.505>form <00:13:09.785>a <00:13:10.065>batch <00:13:10.345>together.

210
00:13:11.495 --> 00:13:14.755
<00:13:11.495>And <00:13:11.622>the <00:13:11.749>default <00:13:11.876>behavior <00:13:12.004>is <00:13:12.131>to <00:13:12.258>hash <00:13:12.385>the <00:13:12.513>key <00:13:12.640>of <00:13:12.767>a <00:13:12.895>record

211
00:13:14.895 --> 00:13:16.275
<00:13:14.895>to <00:13:15.205>get <00:13:15.516>the <00:13:15.826>partition <00:13:16.137>number.

212
00:13:17.455 --> 00:13:20.715
<00:13:17.455>And, <00:13:18.155>um, <00:13:18.855>yeah,

213
00:13:20.735 --> 00:13:24.875
<00:13:20.735>but <00:13:20.862>some, <00:13:20.989>uh, <00:13:21.116>records <00:13:21.244>may <00:13:21.371>have <00:13:21.498>an <00:13:21.625>all <00:13:21.753>key, <00:13:21.880>in <00:13:22.007>this <00:13:22.135>case,

214
00:13:24.895 --> 00:13:27.755
<00:13:24.895>the <00:13:25.245>old <00:13:25.595>partition <00:13:25.944>strategy, <00:13:26.295>uh,

215
00:13:28.735 --> 00:13:32.035
<00:13:28.735>before <00:13:28.935>Apache <00:13:29.135>Kafka <00:13:29.335>2.4 <00:13:29.535>would <00:13:29.735>be <00:13:29.935>to <00:13:30.135>cycle

216
00:13:32.035 --> 00:13:36.275
<00:13:32.035>through <00:13:32.175>the <00:13:32.314>topics <00:13:32.454>partition <00:13:32.594>and <00:13:32.735>sent <00:13:32.875>a <00:13:33.015>record <00:13:33.155>to <00:13:33.295>each <00:13:33.435>one.

217
00:13:37.565 --> 00:13:41.705
<00:13:37.565>So <00:13:37.720>unfortunately, <00:13:37.876>this <00:13:38.031>method <00:13:38.187>does <00:13:38.342>not <00:13:38.498>batch, <00:13:38.653>uh, <00:13:38.809>very <00:13:38.965>well.

218
00:13:42.005 --> 00:13:46.345
<00:13:42.005>And <00:13:42.160>in <00:13:42.316>may, <00:13:42.471>uh, <00:13:42.627>and <00:13:42.782>may <00:13:42.938>in <00:13:43.093>fact <00:13:43.249>add <00:13:43.405>latency.

219
00:13:48.755 --> 00:13:50.815
<00:13:48.755>So <00:13:48.988>this <00:13:49.221>ticket <00:13:49.455>partition <00:13:49.688>addresses <00:13:49.921>the <00:13:50.155>problem

220
00:13:50.995 --> 00:13:55.295
<00:13:50.995>of <00:13:51.170>spreading <00:13:51.345>out <00:13:51.520>records <00:13:51.695>without <00:13:51.870>keys <00:13:52.045>into <00:13:52.220>smaller <00:13:52.395>batches

221
00:13:55.555 --> 00:13:59.025
<00:13:55.555>by <00:13:55.755>picking <00:13:55.954>a <00:13:56.155>single <00:13:56.354>partition <00:13:56.555>to <00:13:56.755>send <00:13:56.954>all

222
00:13:59.565 --> 00:14:00.865
<00:13:59.565>non <00:14:00.150>keyed <00:14:00.735>records.

223
00:14:02.125 --> 00:14:05.985
<00:14:02.125>And <00:14:02.280>once <00:14:02.436>the <00:14:02.591>batch <00:14:02.747>at <00:14:02.902>that <00:14:03.058>partition <00:14:03.213>is <00:14:03.369>filled <00:14:03.525>or

224
00:14:05.985 --> 00:14:08.725
<00:14:05.985>otherwise <00:14:07.385>completed,

225
00:14:09.065 --> 00:14:11.365
<00:14:09.065>the <00:14:09.415>stick <00:14:09.765>partition <00:14:10.115>randomly <00:14:10.465>chooses

226
00:14:11.385 --> 00:14:13.925
<00:14:11.385>and <00:14:11.618>sticks, <00:14:11.851>uh, <00:14:12.085>to <00:14:12.318>a <00:14:12.551>new <00:14:12.785>partition

227
00:14:15.675 --> 00:14:20.485
<00:14:15.675>That <00:14:15.908>way <00:14:16.141>over <00:14:16.375>an <00:14:16.608>extended <00:14:16.841>period, <00:14:17.074>uh,

228
00:14:20.485 --> 00:14:24.685
<00:14:20.485>records <00:14:20.765>are, <00:14:21.045>uh, <00:14:21.325>about <00:14:21.605>evenly <00:14:21.885>distributed

229
00:14:25.135 --> 00:14:29.725
<00:14:25.135>among <00:14:25.415>all <00:14:25.694>the <00:14:25.975>partitions <00:14:26.255>while <00:14:26.535>getting

230
00:14:29.755 --> 00:14:32.605
<00:14:29.755>that <00:14:30.035>benefit <00:14:30.314>of <00:14:30.595>larger <00:14:30.875>batch <00:14:31.155>sizes.

231
00:14:35.435 --> 00:14:40.055
<00:14:35.435>And, <00:14:35.635>uh, <00:14:35.834>when <00:14:36.035>comparing <00:14:36.234>the <00:14:36.435>99 <00:14:36.635>percentile, <00:14:36.834>uh,

232
00:14:40.165 --> 00:14:43.655
<00:14:40.165>latency <00:14:40.320>of <00:14:40.476>a <00:14:40.631>cluster <00:14:40.787>with <00:14:40.942>three <00:14:41.098>producers, <00:14:41.253>uh, <00:14:41.409>that <00:14:41.564>produced

233
00:14:44.215 --> 00:14:46.535
<00:14:44.215>thousand <00:14:44.415>messages <00:14:44.615>per <00:14:44.815>se <00:14:45.015>per <00:14:45.215>second <00:14:45.415>to <00:14:45.615>topics

234
00:14:46.565 --> 00:14:47.735
<00:14:46.565>with <00:14:47.091>16 <00:14:47.618>partitions.

235
00:14:49.275 --> 00:14:51.525
<00:14:49.275>Uh, <00:14:49.555>the <00:14:49.834>sticky <00:14:50.115>partition <00:14:50.395>strategy <00:14:50.675>had

236
00:14:51.525 --> 00:14:53.765
<00:14:51.525>around <00:14:51.725>half <00:14:51.925>the <00:14:52.125>latency <00:14:52.324>of <00:14:52.525>the <00:14:52.725>default <00:14:52.925>strategy.

237
00:14:54.305 --> 00:14:58.645
<00:14:54.305>And <00:14:54.432>you <00:14:54.559>can <00:14:54.686>see <00:14:54.814>test <00:14:54.941>results <00:14:55.068>on <00:14:55.195>the <00:14:55.323>diagrams <00:14:55.450>on <00:14:55.577>the <00:14:55.704>slide.

238
00:15:00.345 --> 00:15:04.695
<00:15:00.345>Uh, <00:15:00.500>and <00:15:00.656>the <00:15:00.811>decrease <00:15:00.967>in <00:15:01.122>latency <00:15:01.278>became <00:15:01.433>even <00:15:01.589>more <00:15:01.745>visible

239
00:15:04.915 --> 00:15:09.295
<00:15:04.915>as <00:15:05.194>partitions <00:15:05.474>increased, <00:15:05.755>uh, <00:15:06.035>which <00:15:06.314>is

240
00:15:10.095 --> 00:15:11.575
<00:15:10.095>actually <00:15:10.361>in <00:15:10.627>line <00:15:10.894>with <00:15:11.160>the <00:15:11.427>idea

241
00:15:11.605 --> 00:15:15.655
<00:15:11.605>that <00:15:11.780>a <00:15:11.955>few <00:15:12.130>larger <00:15:12.305>batches <00:15:12.480>result <00:15:12.655>in <00:15:12.830>lower <00:15:13.005>latency,

242
00:15:16.515 --> 00:15:18.135
<00:15:16.515>uh, <00:15:16.865>than <00:15:17.215>many <00:15:17.564>small <00:15:17.915>batches.

243
00:15:29.585 --> 00:15:30.725
<00:15:29.585>So <00:15:29.841>yeah, <00:15:30.098>that's <00:15:30.354>the <00:15:30.611>slide.

244
00:15:37.835 --> 00:15:39.495
<00:15:37.835>So <00:15:38.301>optimizing <00:15:38.768>for <00:15:39.235>throughput.

245
00:15:41.895 --> 00:15:44.235
<00:15:41.895>So <00:15:42.095>we <00:15:42.295>discussed <00:15:42.495>how <00:15:42.694>to <00:15:42.895>optimize <00:15:43.095>for <00:15:43.295>latency,

246
00:15:44.895 --> 00:15:47.395
<00:15:44.895>and <00:15:45.095>uh, <00:15:45.295>now <00:15:45.495>we're <00:15:45.694>going <00:15:45.895>to <00:15:46.095>optimize, <00:15:46.295>uh,

247
00:15:47.415 --> 00:15:49.115
<00:15:47.415>to <00:15:47.765>discuss <00:15:48.115>optimizing <00:15:48.464>for <00:15:48.814>throughput.

248
00:15:49.855 --> 00:15:52.035
<00:15:49.855>And <00:15:50.055>to <00:15:50.255>optimize <00:15:50.455>for <00:15:50.655>throughput, <00:15:50.855>the <00:15:51.055>producers, <00:15:51.255>brokers

249
00:15:52.035 --> 00:15:56.675
<00:15:52.035>and <00:15:52.162>consumers <00:15:52.289>need <00:15:52.416>to <00:15:52.544>move <00:15:52.671>as <00:15:52.798>much <00:15:52.925>data <00:15:53.053>as <00:15:53.180>they <00:15:53.307>can, <00:15:53.435>uh,

250
00:15:56.675 --> 00:15:58.195
<00:15:56.675>within <00:15:56.948>a <00:15:57.222>given <00:15:57.495>amount <00:15:57.769>of <00:15:58.043>time.

251
00:15:58.775 --> 00:16:01.115
<00:15:58.775>And <00:15:59.008>for <00:15:59.241>high <00:15:59.475>throughput, <00:15:59.708>you <00:15:59.941>are <00:16:00.175>trying

252
00:16:01.115 --> 00:16:04.275
<00:16:01.115>to <00:16:01.315>maximize <00:16:01.515>the <00:16:01.715>rate <00:16:01.915>at <00:16:02.115>which <00:16:02.315>data <00:16:02.515>moves.

253
00:16:06.055 --> 00:16:10.355
<00:16:06.055>And, <00:16:06.194>uh, <00:16:06.334>topic <00:16:06.474>partition <00:16:06.614>is <00:16:06.755>a <00:16:06.895>unit <00:16:07.035>of <00:16:07.175>parallelism <00:16:07.314>in <00:16:07.454>Kafka.

254
00:16:10.495 --> 00:16:14.515
<00:16:10.495>And <00:16:10.650>messages <00:16:10.806>to <00:16:10.961>different <00:16:11.117>partitions <00:16:11.272>can <00:16:11.428>be <00:16:11.583>sent <00:16:11.739>in <00:16:11.895>parallel

255
00:16:14.575 --> 00:16:18.515
<00:16:14.575>by <00:16:14.775>producers <00:16:14.975>written <00:16:15.175>in <00:16:15.375>parallel <00:16:15.575>by, <00:16:15.775>um, <00:16:15.975>brokers

256
00:16:19.775 --> 00:16:22.235
<00:16:19.775>and <00:16:20.055>read <00:16:20.334>in <00:16:20.615>parallel <00:16:20.895>by <00:16:21.175>consumers.

257
00:16:23.535 --> 00:16:27.075
<00:16:23.535>So <00:16:23.709>in <00:16:23.885>general, <00:16:24.060>a <00:16:24.235>high <00:16:24.410>number <00:16:24.584>of <00:16:24.760>topic <00:16:24.935>partitions

258
00:16:27.625 --> 00:16:29.795
<00:16:27.625>results <00:16:28.091>in <00:16:28.558>high <00:16:29.025>throughput.

259
00:16:30.775 --> 00:16:33.515
<00:16:30.775>And <00:16:30.975>to <00:16:31.175>maximize <00:16:31.375>throughput, <00:16:31.574>you <00:16:31.775>want <00:16:31.975>enough <00:16:32.175>partitions

260
00:16:33.615 --> 00:16:36.355
<00:16:33.615>to <00:16:33.815>utilize <00:16:34.015>all <00:16:34.215>brokers <00:16:34.415>in <00:16:34.615>the <00:16:34.815>cluster <00:16:35.015>time.

261
00:16:38.765 --> 00:16:43.605
<00:16:38.765>Uh, <00:16:38.920>but <00:16:39.076>there <00:16:39.231>are <00:16:39.387>trade-offs, <00:16:39.542>uh, <00:16:39.698>to <00:16:39.853>increase <00:16:40.009>the <00:16:40.165>number

262
00:16:43.605 --> 00:16:47.805
<00:16:43.605>of <00:16:43.838>partitions, <00:16:44.071>like <00:16:44.305>a <00:16:44.538>higher <00:16:44.771>request <00:16:45.005>rate

263
00:16:47.905 --> 00:16:52.735
<00:16:47.905>or <00:16:48.138>follow <00:16:48.371>up <00:16:48.605>fetching, <00:16:48.838>uh, <00:16:49.071>which, <00:16:49.305>um,

264
00:16:54.075 --> 00:16:56.725
<00:16:54.075>results <00:16:54.355>in <00:16:54.635>higher <00:16:54.915>broker <00:16:55.195>CPA <00:16:55.475>usage.

265
00:16:58.345 --> 00:17:02.125
<00:16:58.345>So <00:16:58.545>moreover, <00:16:58.745>poor <00:16:58.945>batching <00:16:59.145>might <00:16:59.345>happen <00:16:59.545>when <00:16:59.745>producers

266
00:17:02.635 --> 00:17:05.605
<00:17:02.635>send <00:17:02.810>requests <00:17:02.985>to <00:17:03.160>a <00:17:03.335>very <00:17:03.510>large <00:17:03.685>number <00:17:03.860>of <00:17:04.035>partitions.

267
00:17:06.785 --> 00:17:09.245
<00:17:06.785>So <00:17:07.485>yeah, <00:17:08.185>remember

268
00:17:09.245 --> 00:17:12.765
<00:17:09.244>to <00:17:09.419>choose <00:17:09.594>the <00:17:09.770>partition <00:17:09.944>count <00:17:10.119>carefully <00:17:10.294>based <00:17:10.469>on <00:17:10.645>producer

269
00:17:12.765 --> 00:17:14.485
<00:17:12.765>throughput <00:17:13.231>and <00:17:13.698>consumer <00:17:14.165>throughput,

270
00:17:15.905 --> 00:17:19.125
<00:17:15.905>and, <00:17:16.138>uh, <00:17:16.371>benchmark <00:17:16.605>performance <00:17:16.838>in <00:17:17.071>your <00:17:17.305>environment.

271
00:17:23.475 --> 00:17:26.605
<00:17:23.474>Also <00:17:23.754>take <00:17:24.034>into <00:17:24.314>consideration <00:17:24.594>the <00:17:24.875>design

272
00:17:26.605 --> 00:17:27.685
<00:17:26.605>of <00:17:26.929>your <00:17:27.252>data <00:17:27.577>patterns

273
00:17:27.905 --> 00:17:32.525
<00:17:27.905>and <00:17:28.079>key <00:17:28.254>assignments <00:17:28.430>so <00:17:28.605>that, <00:17:28.780>uh, <00:17:28.955>messages <00:17:29.129>are <00:17:29.305>distributed

274
00:17:32.785 --> 00:17:36.365
<00:17:32.785>as <00:17:32.960>evenly <00:17:33.135>as <00:17:33.310>possible <00:17:33.485>across, <00:17:33.660>uh, <00:17:33.835>the <00:17:34.010>topic <00:17:34.185>partitions.

275
00:17:39.415 --> 00:17:41.915
<00:17:39.415>Um, <00:17:39.764>'cause <00:17:40.115>it'll <00:17:40.465>prevent, <00:17:40.815>uh,

276
00:17:41.915 --> 00:17:45.355
<00:17:41.915>certain <00:17:42.148>topic <00:17:42.381>partitions <00:17:42.615>from <00:17:42.848>getting <00:17:43.081>overloaded <00:17:43.315>relative

277
00:17:45.355 --> 00:17:46.515
<00:17:45.355>to <00:17:45.703>other <00:17:46.051>topic <00:17:46.399>partitions.

278
00:17:47.495 --> 00:17:52.005
<00:17:47.495>And <00:17:47.695>yeah, <00:17:47.895>so <00:17:48.094>yeah, <00:17:48.294>you <00:17:48.495>can, <00:17:48.695>you <00:17:48.895>need

279
00:17:52.005 --> 00:17:54.485
<00:17:52.005>to <00:17:52.145>monitor <00:17:52.285>your <00:17:52.425>data <00:17:52.565>and <00:17:52.705>make <00:17:52.845>sure <00:17:52.985>that <00:17:53.125>it's <00:17:53.265>not <00:17:53.405>skewed.

280
00:17:56.785 --> 00:17:59.725
<00:17:56.785>Uh, <00:17:56.985>the <00:17:57.185>most <00:17:57.385>important <00:17:57.585>step <00:17:57.785>you <00:17:57.985>can <00:17:58.185>take

281
00:17:59.725 --> 00:18:03.005
<00:17:59.725>to <00:17:59.899>optimize <00:18:00.074>throughput <00:18:00.250>is <00:18:00.425>to <00:18:00.600>tune <00:18:00.774>the <00:18:00.949>producer <00:18:01.125>batching

282
00:18:03.005 --> 00:18:04.245
<00:18:03.005>to <00:18:03.284>increase <00:18:03.563>the <00:18:03.841>batch <00:18:04.120>size

283
00:18:04.665 --> 00:18:08.125
<00:18:04.665>and <00:18:04.865>the <00:18:05.065>time <00:18:05.264>spent <00:18:05.465>waiting <00:18:05.665>for <00:18:05.865>the <00:18:06.065>batch

284
00:18:08.145 --> 00:18:09.445
<00:18:08.145>to <00:18:08.437>fill <00:18:08.730>up <00:18:09.022>with <00:18:09.315>messages.

285
00:18:11.105 --> 00:18:13.565
<00:18:11.105>Uh, <00:18:11.305>you <00:18:11.505>can <00:18:11.705>configure <00:18:11.905>the <00:18:12.105>batch <00:18:12.305>size, <00:18:12.505>uh,

286
00:18:13.885 --> 00:18:16.805
<00:18:13.885>configuration <00:18:14.118>parameter <00:18:14.351>to <00:18:14.585>increase <00:18:14.818>the <00:18:15.051>maximum <00:18:15.285>size

287
00:18:16.805 --> 00:18:18.445
<00:18:16.805>and <00:18:17.085>bytes <00:18:17.365>of <00:18:17.645>each <00:18:17.925>message <00:18:18.205>batch,

288
00:18:19.745 --> 00:18:22.205
<00:18:19.745>and <00:18:19.919>to <00:18:20.094>give <00:18:20.270>more <00:18:20.445>time <00:18:20.620>for <00:18:20.794>batches <00:18:20.969>to <00:18:21.145>fill.

289
00:18:22.205 --> 00:18:24.765
<00:18:22.205>You <00:18:22.438>can <00:18:22.671>configure <00:18:22.905>the <00:18:23.138>linger <00:18:23.371>mass <00:18:23.605>parameter

290
00:18:24.785 --> 00:18:26.605
<00:18:24.785>to <00:18:25.065>have <00:18:25.345>the <00:18:25.625>producer <00:18:25.905>wait <00:18:26.185>longer

291
00:18:26.745 --> 00:18:31.565
<00:18:26.745>before <00:18:26.945>sending, <00:18:27.145>uh, <00:18:27.344>the <00:18:27.544>delay <00:18:27.745>allows <00:18:27.945>the <00:18:28.145>producer

292
00:18:31.705 --> 00:18:33.645
<00:18:31.705>to <00:18:31.905>wait <00:18:32.105>for <00:18:32.304>the <00:18:32.504>batch <00:18:32.705>to <00:18:32.905>reach, <00:18:33.105>uh,

293
00:18:33.645 --> 00:18:35.005
<00:18:33.645>the <00:18:34.053>configured <00:18:34.461>batch <00:18:34.869>size.

294
00:18:36.585 --> 00:18:40.045
<00:18:36.585>Uh, <00:18:36.760>but, <00:18:36.935>uh, <00:18:37.110>there <00:18:37.285>is <00:18:37.460>a <00:18:37.635>trade <00:18:37.810>off, <00:18:37.985>uh,

295
00:18:40.045 --> 00:18:42.725
<00:18:40.045>which <00:18:40.325>is <00:18:40.605>tolerating <00:18:40.885>higher <00:18:41.165>latency, <00:18:41.445>uh,

296
00:18:42.775 --> 00:18:45.485
<00:18:42.775>since <00:18:43.008>messages <00:18:43.241>are <00:18:43.475>not <00:18:43.708>sent <00:18:43.941>as <00:18:44.175>soon

297
00:18:45.485 --> 00:18:47.005
<00:18:45.485>as <00:18:45.758>they're <00:18:46.032>ready <00:18:46.305>to <00:18:46.579>be <00:18:46.853>sent.

298
00:18:49.815 --> 00:18:53.065
<00:18:49.815>Also, <00:18:49.970>you <00:18:50.126>can <00:18:50.281>easily <00:18:50.437>enable <00:18:50.592>compression, <00:18:50.748>which <00:18:50.903>means <00:18:51.059>a <00:18:51.215>lot

299
00:18:53.065 --> 00:18:56.185
<00:18:53.065>of <00:18:53.240>bits <00:18:53.415>can <00:18:53.590>be <00:18:53.765>sent <00:18:53.940>as <00:18:54.115>fewer <00:18:54.290>bits, <00:18:54.465>basically.

300
00:18:57.445 --> 00:18:59.265
<00:18:57.445>And <00:18:57.725>yeah, <00:18:58.004>you <00:18:58.284>can <00:18:58.564>enable <00:18:58.845>compression

301
00:18:59.265 --> 00:19:00.985
<00:18:59.265>by <00:18:59.615>configuring <00:18:59.965>the <00:19:00.315>compression <00:19:00.665>type

302
00:19:01.405 --> 00:19:02.705
<00:19:01.405>or <00:19:01.990>configuration <00:19:02.574>parameter.

303
00:19:04.805 --> 00:19:07.585
<00:19:04.805>And, <00:19:05.005>uh, <00:19:05.205>there <00:19:05.405>are <00:19:05.605>four <00:19:05.805>standard <00:19:06.005>compression <00:19:06.205>codex

304
00:19:08.245 --> 00:19:10.945
<00:19:08.245>LZ <00:19:08.478>four <00:19:08.711>snipes <00:19:08.945>at <00:19:09.178>STD <00:19:09.411>and <00:19:09.645>JZ.

305
00:19:12.005 --> 00:19:15.505
<00:19:12.005>And <00:19:12.205>for <00:19:12.405>performance, <00:19:12.605>we <00:19:12.805>generally <00:19:13.005>recommend <00:19:13.205>avoiding <00:19:13.405>JZ

306
00:19:15.505 --> 00:19:17.185
<00:19:15.505>because <00:19:15.785>it <00:19:16.065>can <00:19:16.345>be <00:19:16.625>CP <00:19:16.905>intensive.

307
00:19:19.745 --> 00:19:23.815
<00:19:19.745>And, <00:19:20.094>uh, <00:19:20.445>compression <00:19:20.794>is <00:19:21.145>applied

308
00:19:23.915 --> 00:19:25.135
<00:19:23.915>on <00:19:24.189>full <00:19:24.464>batches <00:19:24.738>of <00:19:25.013>data.

309
00:19:25.595 --> 00:19:26.855
<00:19:25.595>So <00:19:26.162>the <00:19:26.729>efficiency

310
00:19:26.855 --> 00:19:29.455
<00:19:26.855>of <00:19:27.055>batching <00:19:27.255>will <00:19:27.455>also <00:19:27.655>impact <00:19:27.855>the <00:19:28.055>compression <00:19:28.255>ratio.

311
00:19:31.505 --> 00:19:36.295
<00:19:31.505>Uh, <00:19:31.971>important <00:19:32.438>thing <00:19:32.905>is,

312
00:19:36.875 --> 00:19:38.295
<00:19:36.875>uh, <00:19:37.130>you <00:19:37.386>need <00:19:37.641>to <00:19:37.897>make <00:19:38.153>sure

313
00:19:38.295 --> 00:19:40.735
<00:19:38.295>that <00:19:38.528>compression <00:19:38.761>codex <00:19:38.995>are <00:19:39.228>matching <00:19:39.461>for <00:19:39.695>blue

314
00:19:40.835 --> 00:19:42.135
<00:19:40.835>for <00:19:41.225>producers <00:19:41.615>and <00:19:42.005>topics.

315
00:19:44.355 --> 00:19:47.255
<00:19:44.355>Uh, <00:19:44.588>otherwise <00:19:44.821>the <00:19:45.055>broker <00:19:45.288>will <00:19:45.521>recompress <00:19:45.755>data

316
00:19:47.395 --> 00:19:49.375
<00:19:47.395>to <00:19:47.628>match <00:19:47.861>the <00:19:48.095>code, <00:19:48.328>code <00:19:48.561>destination <00:19:48.795>topic,

317
00:19:49.625 --> 00:19:51.615
<00:19:49.625>which <00:19:49.858>can <00:19:50.091>result <00:19:50.325>in <00:19:50.558>a <00:19:50.791>performance <00:19:51.025>impact.

318
00:19:53.655 --> 00:19:57.035
<00:19:53.655>And <00:19:53.782>when <00:19:53.909>a <00:19:54.036>producer <00:19:54.164>sends <00:19:54.291>a <00:19:54.418>message <00:19:54.545>to <00:19:54.673>a <00:19:54.800>Kafka <00:19:54.927>cluster, <00:19:55.055>uh,

319
00:19:57.135 --> 00:19:58.835
<00:19:57.135>the <00:19:57.335>message <00:19:57.535>is <00:19:57.735>sent <00:19:57.935>to <00:19:58.135>the <00:19:58.335>leader <00:19:58.535>broker

320
00:19:58.935 --> 00:20:00.075
<00:19:58.935>for <00:19:59.277>the <00:19:59.619>target <00:19:59.961>partition,

321
00:20:00.895 --> 00:20:03.955
<00:20:00.895>and <00:20:01.050>then <00:20:01.206>the <00:20:01.361>producer <00:20:01.517>awaits <00:20:01.672>a <00:20:01.828>response <00:20:01.983>from <00:20:02.139>the <00:20:02.295>leader

322
00:20:03.955 --> 00:20:08.435
<00:20:03.955>broker <00:20:04.110>to <00:20:04.266>know <00:20:04.421>that, <00:20:04.577>uh, <00:20:04.732>its <00:20:04.888>message <00:20:05.043>has <00:20:05.199>been <00:20:05.355>committed

323
00:20:09.095 --> 00:20:12.835
<00:20:09.095>before <00:20:09.328>proceeding <00:20:09.561>to <00:20:09.795>send, <00:20:10.028>um, <00:20:10.261>next <00:20:10.495>event.

324
00:20:14.735 --> 00:20:16.875
<00:20:14.735>And <00:20:14.968>the <00:20:15.201>sooner <00:20:15.435>producer <00:20:15.668>receives <00:20:15.901>a <00:20:16.135>response,

325
00:20:16.895 --> 00:20:18.915
<00:20:16.895>the <00:20:17.070>sooner <00:20:17.245>the <00:20:17.420>producer <00:20:17.595>can <00:20:17.770>send <00:20:17.945>the <00:20:18.120>next <00:20:18.295>message,

326
00:20:19.125 --> 00:20:21.915
<00:20:19.125>which <00:20:19.405>general <00:20:19.685>results <00:20:19.965>in <00:20:20.245>higher <00:20:20.525>throughput.

327
00:20:23.335 --> 00:20:25.475
<00:20:23.335>And, <00:20:23.685>uh, <00:20:24.035>so <00:20:24.385>setting <00:20:24.735>X

328
00:20:25.695 --> 00:20:29.235
<00:20:25.695>to <00:20:25.835>one <00:20:25.975>makes <00:20:26.115>the <00:20:26.254>leader <00:20:26.395>broker <00:20:26.534>write <00:20:26.675>record <00:20:26.814>to <00:20:26.955>its <00:20:27.095>local,

329
00:20:29.285 --> 00:20:31.515
<00:20:29.285>local <00:20:29.518>only, <00:20:29.751>and <00:20:29.985>then <00:20:30.218>acknowledge <00:20:30.451>the <00:20:30.685>request

330
00:20:32.025 --> 00:20:34.515
<00:20:32.025>without <00:20:32.305>awaiting <00:20:32.585>acknowledgement <00:20:32.865>from <00:20:33.145>all <00:20:33.425>followers.

331
00:20:35.775 --> 00:20:38.315
<00:20:35.775>Uh, <00:20:35.975>but <00:20:36.175>there <00:20:36.375>are <00:20:36.575>trade <00:20:36.775>offs <00:20:36.975>again, <00:20:37.175>uh,

332
00:20:38.575 --> 00:20:42.075
<00:20:38.575>the <00:20:38.730>trade <00:20:38.886>off <00:20:39.041>is <00:20:39.197>you <00:20:39.352>have <00:20:39.508>to <00:20:39.663>tolerate <00:20:39.819>lower <00:20:39.975>durability

333
00:20:42.075 --> 00:20:45.355
<00:20:42.075>because <00:20:42.250>the <00:20:42.425>producer, <00:20:42.600>uh, <00:20:42.775>does <00:20:42.950>not <00:20:43.125>have <00:20:43.300>to <00:20:43.475>wait

334
00:20:45.485 --> 00:20:49.125
<00:20:45.485>until <00:20:45.685>the <00:20:45.885>message <00:20:46.084>is <00:20:46.284>replicated <00:20:46.485>to <00:20:46.685>other <00:20:46.885>brokers.

335
00:20:52.835 --> 00:20:55.815
<00:20:52.835>Uh, <00:20:53.068>likewise <00:20:53.301>you <00:20:53.535>can <00:20:53.768>tune <00:20:54.001>the <00:20:54.235>consumers

336
00:20:55.835 --> 00:20:57.495
<00:20:55.835>for <00:20:56.185>higher <00:20:56.535>throughput <00:20:56.885>by <00:20:57.235>adjusting

337
00:20:57.515 --> 00:21:00.615
<00:20:57.515>how <00:20:57.655>much <00:20:57.795>data <00:20:57.935>it <00:20:58.075>gets <00:20:58.215>from <00:20:58.355>each <00:20:58.495>fetch <00:20:58.635>from <00:20:58.775>the <00:20:58.915>leader

338
00:21:00.615 --> 00:21:03.855
<00:21:00.615>broker, <00:21:00.965>and <00:21:01.315>you <00:21:01.665>can <00:21:02.015>increase

339
00:21:03.915 --> 00:21:06.375
<00:21:03.915>how <00:21:04.090>much <00:21:04.264>data <00:21:04.440>the <00:21:04.615>consumers <00:21:04.790>get <00:21:04.965>from <00:21:05.139>the <00:21:05.315>leader

340
00:21:06.435 --> 00:21:07.615
<00:21:06.435>for <00:21:06.789>each <00:21:07.143>fetch <00:21:07.497>request.

341
00:21:07.955 --> 00:21:10.215
<00:21:07.955>Uh, <00:21:08.235>by <00:21:08.514>increasing <00:21:08.794>the <00:21:09.074>configuration <00:21:09.355>parameter,

342
00:21:10.305 --> 00:21:11.335
<00:21:10.305>fetch <00:21:10.768>mean <00:21:11.232>bytes.

343
00:21:12.895 --> 00:21:16.275
<00:21:12.895>And <00:21:13.095>this <00:21:13.295>parameter <00:21:13.495>basically <00:21:13.695>sets <00:21:13.895>the <00:21:14.095>minimum <00:21:14.295>number

344
00:21:16.335 --> 00:21:20.035
<00:21:16.335>of <00:21:16.490>bytes <00:21:16.646>expected <00:21:16.801>for <00:21:16.957>a <00:21:17.112>fetch <00:21:17.268>response <00:21:17.423>from <00:21:17.579>a <00:21:17.735>consumer.

345
00:21:21.365 --> 00:21:23.745
<00:21:21.365>And <00:21:21.565>increasing <00:21:21.765>this <00:21:21.965>will <00:21:22.165>also <00:21:22.365>reduce <00:21:22.565>the <00:21:22.765>number

346
00:21:23.745 --> 00:21:25.905
<00:21:23.745>of <00:21:23.978>fetch <00:21:24.211>requests <00:21:24.445>made <00:21:24.678>to <00:21:24.911>the <00:21:25.145>broker,

347
00:21:26.545 --> 00:21:30.325
<00:21:26.545>reducing <00:21:26.825>the <00:21:27.105>broker <00:21:27.385>CPO <00:21:27.665>overhead, <00:21:27.945>uh,

348
00:21:30.985 --> 00:21:33.365
<00:21:30.985>to <00:21:31.334>process <00:21:31.685>each <00:21:32.034>fetch <00:21:32.385>request,

349
00:21:33.745 --> 00:21:36.805
<00:21:33.745>and <00:21:34.024>they <00:21:34.304>thereby <00:21:34.584>also <00:21:34.864>improving <00:21:35.145>throughput.

350
00:21:40.755 --> 00:21:40.975
Uh,

351
00:21:54.405 --> 00:21:55.405
<00:21:54.405>So <00:21:54.855>low <00:21:55.305>testing.

352
00:21:56.325 --> 00:21:58.865
<00:21:56.325>How <00:21:56.480>do <00:21:56.636>you <00:21:56.791>relate <00:21:56.947>performance <00:21:57.102>for <00:21:57.258>all <00:21:57.413>different <00:21:57.569>use <00:21:57.725>cases?

353
00:21:59.925 --> 00:22:02.065
<00:21:59.925>Uh, <00:22:00.100>well, <00:22:00.274>you <00:22:00.450>need <00:22:00.625>to <00:22:00.800>measure, <00:22:00.975>measure <00:22:01.149>and <00:22:01.325>measure.

354
00:22:02.445 --> 00:22:04.065
<00:22:02.445>And, <00:22:02.794>uh, <00:22:03.145>whenever <00:22:03.495>you <00:22:03.845>want

355
00:22:04.065 --> 00:22:05.945
<00:22:04.065>to <00:22:04.298>change <00:22:04.531>any <00:22:04.765>configuration <00:22:04.998>parameter, <00:22:05.231>you <00:22:05.465>need

356
00:22:05.945 --> 00:22:07.785
<00:22:05.945>to <00:22:06.225>run <00:22:06.504>performance <00:22:06.784>tests <00:22:07.064>to <00:22:07.345>understand

357
00:22:07.845 --> 00:22:09.185
<00:22:07.845>how <00:22:08.146>it <00:22:08.448>affects <00:22:08.749>your <00:22:09.051>system.

358
00:22:10.765 --> 00:22:13.065
<00:22:10.765>Uh, <00:22:10.940>for <00:22:11.115>example, <00:22:11.290>you <00:22:11.465>want <00:22:11.640>to <00:22:11.815>enable <00:22:11.990>the <00:22:12.165>events,

359
00:22:13.445 --> 00:22:14.465
<00:22:13.445>uh, <00:22:14.362>compression.

360
00:22:15.965 --> 00:22:18.825
<00:22:15.965>Uh, <00:22:16.165>you <00:22:16.365>need <00:22:16.564>to <00:22:16.764>run <00:22:16.965>multiple <00:22:17.165>benchmarking <00:22:17.365>tests

361
00:22:18.925 --> 00:22:21.585
<00:22:18.925>to <00:22:19.100>understand <00:22:19.274>if <00:22:19.450>there's <00:22:19.625>the <00:22:19.800>right <00:22:19.975>decision <00:22:20.149>for <00:22:20.325>you.

362
00:22:23.595 --> 00:22:26.255
<00:22:23.595>Uh, <00:22:23.770>law <00:22:23.945>testing <00:22:24.120>your <00:22:24.295>system <00:22:24.470>has <00:22:24.645>three <00:22:24.820>main <00:22:24.995>objectives

363
00:22:28.035 --> 00:22:31.415
<00:22:28.035>and <00:22:28.235>they, <00:22:28.435>uh, <00:22:28.635>assess <00:22:28.835>the <00:22:29.035>system's <00:22:29.235>breaking <00:22:29.435>points,

364
00:22:31.675 --> 00:22:33.895
<00:22:31.675>assess <00:22:32.024>the <00:22:32.375>system's <00:22:32.725>maximum <00:22:33.075>capacity

365
00:22:34.235 --> 00:22:37.535
<00:22:34.235>and <00:22:34.435>verify <00:22:34.635>the <00:22:34.834>system's <00:22:35.034>ability <00:22:35.235>to <00:22:35.435>recover <00:22:35.635>afterwards.

366
00:22:39.115 --> 00:22:41.655
<00:22:39.115>And <00:22:39.465>you <00:22:39.815>can <00:22:40.165>test <00:22:40.515>these

367
00:22:42.575 --> 00:22:45.615
<00:22:42.575>measures <00:22:42.775>in <00:22:42.975>three <00:22:43.175>different <00:22:43.375>ways, <00:22:43.575>uh, <00:22:43.775>which <00:22:43.975>are

368
00:22:46.315 --> 00:22:48.695
<00:22:46.315>stress <00:22:47.015>testing <00:22:47.715>or

369
00:22:48.945 --> 00:22:52.935
<00:22:48.945>where <00:22:49.100>you <00:22:49.256>put <00:22:49.411>your <00:22:49.567>system <00:22:49.722>under <00:22:49.878>heavy <00:22:50.033>load, <00:22:50.189>uh, <00:22:50.345>just

370
00:22:52.955 --> 00:22:57.655
<00:22:52.955>beyond <00:22:53.304>average <00:22:53.655>operational <00:22:54.004>capacity, <00:22:54.355>often

371
00:22:57.755 --> 00:23:00.535
<00:22:57.755>to <00:22:57.955>breaking <00:22:58.155>point <00:22:58.355>and <00:22:58.555>you <00:22:58.755>observe <00:22:58.955>the <00:22:59.155>results.

372
00:23:01.195 --> 00:23:04.535
<00:23:01.195>And <00:23:01.428>then <00:23:01.661>spike <00:23:01.895>testing <00:23:02.128>is <00:23:02.361>a <00:23:02.595>type

373
00:23:04.535 --> 00:23:09.375
<00:23:04.535>of <00:23:04.690>performance <00:23:04.846>testing <00:23:05.001>in <00:23:05.157>which <00:23:05.312>a <00:23:05.468>system <00:23:05.623>receives <00:23:05.779>a <00:23:05.935>sudden

374
00:23:09.475 --> 00:23:11.335
<00:23:09.475>and <00:23:09.824>extreme <00:23:10.175>increase <00:23:10.524>in <00:23:10.875>load.

375
00:23:11.955 --> 00:23:14.815
<00:23:11.955>And <00:23:12.095>the <00:23:12.235>goal <00:23:12.375>of <00:23:12.514>spike <00:23:12.655>testing <00:23:12.794>is <00:23:12.935>to <00:23:13.074>determine <00:23:13.215>the <00:23:13.355>behavior

376
00:23:14.875 --> 00:23:18.095
<00:23:14.875>of <00:23:15.030>a <00:23:15.186>system <00:23:15.341>when <00:23:15.497>it <00:23:15.652>receives <00:23:15.808>extreme <00:23:15.963>variations <00:23:16.119>in <00:23:16.275>traffic.

377
00:23:20.105 --> 00:23:23.165
<00:23:20.105>And, <00:23:20.385>uh, <00:23:20.665>finally, <00:23:20.945>so <00:23:21.225>testing, <00:23:21.505>uh,

378
00:23:23.165 --> 00:23:25.325
<00:23:23.165>checks <00:23:23.445>the <00:23:23.725>endurance <00:23:24.004>of <00:23:24.284>your <00:23:24.565>system.

379
00:23:27.065 --> 00:23:29.245
<00:23:27.065>It <00:23:27.415>involves <00:23:27.765>testing <00:23:28.115>a <00:23:28.465>system

380
00:23:29.275 --> 00:23:32.285
<00:23:29.275>with <00:23:29.475>a <00:23:29.675>topical <00:23:29.875>production <00:23:30.075>load <00:23:30.275>over <00:23:30.475>a <00:23:30.675>continuous

381
00:23:32.285 --> 00:23:35.365
<00:23:32.285>availability <00:23:32.565>period <00:23:32.845>to <00:23:33.125>validate <00:23:33.405>system <00:23:33.685>behavior

382
00:23:35.455 --> 00:23:36.765
<00:23:35.455>under <00:23:36.044>production <00:23:36.634>use.

383
00:23:40.015 --> 00:23:42.515
<00:23:40.015>And <00:23:40.190>when <00:23:40.365>you <00:23:40.540>run <00:23:40.715>your <00:23:40.890>performance <00:23:41.065>tests, <00:23:41.240>you <00:23:41.415>need

384
00:23:42.515 --> 00:23:44.275
<00:23:42.515>to <00:23:42.748>pay <00:23:42.981>attention <00:23:43.215>to <00:23:43.448>system <00:23:43.681>set <00:23:43.915>up.

385
00:23:45.225 --> 00:23:48.845
<00:23:45.225>So <00:23:45.399>make <00:23:45.574>sure <00:23:45.750>you <00:23:45.925>do <00:23:46.100>not <00:23:46.274>run <00:23:46.449>performance <00:23:46.625>tests,

386
00:23:50.625 --> 00:23:54.005
<00:23:50.625>uh, <00:23:50.765>from <00:23:50.905>the <00:23:51.045>same <00:23:51.185>machine <00:23:51.325>as <00:23:51.465>you <00:23:51.605>host <00:23:51.745>your <00:23:51.885>Kafka <00:23:52.025>broker

387
00:23:54.305 --> 00:23:56.085
<00:23:54.305>or <00:23:54.655>any <00:23:55.005>other <00:23:55.355>Kafka <00:23:55.705>component.

388
00:23:58.145 --> 00:24:02.355
<00:23:58.145>Uh, <00:23:58.300>also <00:23:58.456>make <00:23:58.611>sure <00:23:58.767>that <00:23:58.922>there <00:23:59.078>is <00:23:59.233>enough <00:23:59.389>network <00:23:59.545>capacity

389
00:24:02.355 --> 00:24:05.195
<00:24:02.355>between <00:24:02.555>your <00:24:02.755>test <00:24:02.955>cluster <00:24:03.155>and <00:24:03.355>your <00:24:03.555>cover <00:24:03.755>cluster.

390
00:24:07.165 --> 00:24:10.265
<00:24:07.165>Uh, <00:24:07.398>check <00:24:07.631>if <00:24:07.865>it <00:24:08.098>has <00:24:08.331>enough <00:24:08.565>CPU

391
00:24:10.265 --> 00:24:12.425
<00:24:10.265>and <00:24:10.498>memory <00:24:10.731>to <00:24:10.965>run <00:24:11.198>your <00:24:11.431>performance <00:24:11.665>tests.

392
00:24:14.315 --> 00:24:16.695
<00:24:14.315>Pay <00:24:14.595>attention <00:24:14.875>to <00:24:15.155>the <00:24:15.435>system <00:24:15.715>setup

393
00:24:17.325 --> 00:24:20.895
<00:24:17.325>like <00:24:17.605>swap <00:24:17.885>configuration <00:24:18.165>network, <00:24:18.445>I-O-J-V-M <00:24:18.725>settings.

394
00:24:23.655 --> 00:24:28.085
<00:24:23.655>Um, <00:24:24.355>yes, <00:24:25.055>uh,

395
00:24:28.275 --> 00:24:30.885
<00:24:28.275>each <00:24:28.450>system <00:24:28.625>needs <00:24:28.800>time <00:24:28.975>to <00:24:29.150>get <00:24:29.325>up <00:24:29.500>to <00:24:29.675>speed

396
00:24:31.185 --> 00:24:34.925
<00:24:31.185>and <00:24:31.340>for <00:24:31.496>every <00:24:31.651>Java <00:24:31.807>application, <00:24:31.962>uh, <00:24:32.118>GI <00:24:32.273>compile <00:24:32.429>an <00:24:32.585>instrument

397
00:24:34.945 --> 00:24:37.685
<00:24:34.945>to <00:24:35.145>bring <00:24:35.345>it <00:24:35.544>into <00:24:35.745>steady <00:24:35.945>and <00:24:36.145>performance <00:24:36.345>date.

398
00:24:39.385 --> 00:24:41.325
<00:24:39.385>And <00:24:39.665>Kafka <00:24:39.945>is <00:24:40.225>a <00:24:40.504>Java <00:24:40.785>application

399
00:24:41.905 --> 00:24:43.205
<00:24:41.905>and <00:24:42.197>you <00:24:42.490>do <00:24:42.782>not <00:24:43.074>want

400
00:24:43.205 --> 00:24:46.205
<00:24:43.205>to <00:24:43.360>benchmark <00:24:43.516>a <00:24:43.671>new <00:24:43.827>Kafka <00:24:43.982>cluster <00:24:44.138>from <00:24:44.293>its <00:24:44.449>first <00:24:44.605>second,

401
00:24:47.385 --> 00:24:49.125
<00:24:47.385>so <00:24:47.585>it <00:24:47.785>needs <00:24:47.985>some <00:24:48.185>time <00:24:48.385>to <00:24:48.585>warm <00:24:48.785>up

402
00:24:49.305 --> 00:24:51.285
<00:24:49.305>and <00:24:49.655>the <00:24:50.005>Java-based <00:24:50.355>benchmarking <00:24:50.705>tool

403
00:24:51.535 --> 00:24:52.845
<00:24:51.535>needs <00:24:51.770>to <00:24:52.006>warm <00:24:52.242>up <00:24:52.478>as <00:24:52.714>well.

404
00:24:54.255 --> 00:24:57.555
<00:24:54.255>Uh, <00:24:54.410>sometimes <00:24:54.566>you <00:24:54.721>can <00:24:54.877>observe <00:24:55.032>latency <00:24:55.188>spike <00:24:55.343>at <00:24:55.499>the <00:24:55.655>very

405
00:24:57.555 --> 00:25:00.155
<00:24:57.555>beginning, <00:24:57.835>which <00:24:58.115>drops <00:24:58.395>a <00:24:58.675>few <00:24:58.955>second,

406
00:25:00.335 --> 00:25:01.995
<00:25:00.335>so <00:25:00.801>like <00:25:01.268>milliseconds <00:25:01.735>later.

407
00:25:04.295 --> 00:25:07.195
<00:25:04.295>And, <00:25:04.495>uh, <00:25:04.695>when <00:25:04.895>you <00:25:05.095>run <00:25:05.295>performance <00:25:05.495>tests, <00:25:05.695>uh,

408
00:25:07.195 --> 00:25:09.275
<00:25:07.195>there <00:25:07.395>is <00:25:07.595>always <00:25:07.794>some <00:25:07.995>run <00:25:08.195>to <00:25:08.395>wrong <00:25:08.595>variants.

409
00:25:10.175 --> 00:25:13.435
<00:25:10.175>Uh, <00:25:10.375>sometimes <00:25:10.575>your <00:25:10.774>tests <00:25:10.975>are <00:25:11.175>a <00:25:11.375>bit <00:25:11.575>faster,

410
00:25:13.865 --> 00:25:18.465
<00:25:13.865>sometimes <00:25:14.145>they're <00:25:14.425>a <00:25:14.705>bit <00:25:14.985>slower, <00:25:15.265>And

411
00:25:19.945 --> 00:25:23.125
<00:25:19.945>you <00:25:20.145>should <00:25:20.345>run <00:25:20.544>your <00:25:20.745>tests <00:25:20.945>multiple <00:25:21.145>times, <00:25:21.345>uh,

412
00:25:23.985 --> 00:25:26.085
<00:25:23.985>to <00:25:24.264>understand <00:25:24.544>your <00:25:24.824>like <00:25:25.104>actual <00:25:25.385>baseline.

413
00:25:26.425 --> 00:25:30.595
<00:25:26.425>And <00:25:26.580>for <00:25:26.736>comparison, <00:25:26.891>usually <00:25:27.047>one <00:25:27.202>test <00:25:27.358>run <00:25:27.513>is <00:25:27.669>not <00:25:27.825>enough.

414
00:25:32.795 --> 00:25:34.655
<00:25:32.795>And, <00:25:33.028>uh, <00:25:33.261>when <00:25:33.495>you <00:25:33.728>run <00:25:33.961>your <00:25:34.195>test,

415
00:25:34.755 --> 00:25:37.055
<00:25:34.755>the <00:25:34.988>latency <00:25:35.221>runs <00:25:35.455>should <00:25:35.688>be <00:25:35.921>normally <00:25:36.155>distributed.

416
00:25:41.185 --> 00:25:43.995
<00:25:41.185>When <00:25:41.418>you <00:25:41.651>look <00:25:41.885>at <00:25:42.118>your <00:25:42.351>test <00:25:42.585>results,

417
00:25:44.905 --> 00:25:47.075
<00:25:44.905>make <00:25:45.105>sure <00:25:45.305>you <00:25:45.504>don't <00:25:45.705>look <00:25:45.905>at <00:25:46.105>the <00:25:46.305>mean

418
00:25:47.975 --> 00:25:49.795
<00:25:47.975>as <00:25:48.324>the <00:25:48.675>mean <00:25:49.024>is <00:25:49.375>meaningless.

419
00:25:50.855 --> 00:25:53.075
<00:25:50.855>It <00:25:51.135>just, <00:25:51.415>uh, <00:25:51.695>means <00:25:51.975>that <00:25:52.255>half

420
00:25:53.095 --> 00:25:55.715
<00:25:53.095>of <00:25:53.295>the <00:25:53.495>requests <00:25:53.695>were <00:25:53.895>worse <00:25:54.095>than <00:25:54.295>the <00:25:54.495>mean,

421
00:25:57.825 --> 00:26:00.605
<00:25:57.825>and <00:25:58.000>the <00:25:58.175>better <00:25:58.350>solution <00:25:58.525>is <00:25:58.700>to <00:25:58.875>look <00:25:59.050>at <00:25:59.225>percentiles.

422
00:26:01.405 --> 00:26:05.485
<00:26:01.405>A <00:26:01.580>percentile <00:26:01.754>is <00:26:01.930>basically <00:26:02.105>a <00:26:02.280>value <00:26:02.455>below <00:26:02.629>which, <00:26:02.805>uh,

423
00:26:05.635 --> 00:26:09.325
<00:26:05.635>certain <00:26:05.985>percentage <00:26:06.335>of <00:26:06.685>observation <00:26:07.035>lie.

424
00:26:10.385 --> 00:26:13.765
<00:26:10.385>So <00:26:10.585>it <00:26:10.785>is <00:26:10.985>quite <00:26:11.185>common <00:26:11.385>to <00:26:11.585>use <00:26:11.785>95%

425
00:26:14.425 --> 00:26:16.405
<00:26:14.425>or <00:26:15.125>99 <00:26:15.825>percentiles.

426
00:26:17.825 --> 00:26:22.605
<00:26:17.825>If <00:26:17.952>your <00:26:18.079>latency <00:26:18.206>is <00:26:18.334>in <00:26:18.461>99 <00:26:18.588>percentile, <00:26:18.715>it <00:26:18.843>means <00:26:18.970>that <00:26:19.097>only <00:26:19.225>1%

427
00:26:22.745 --> 00:26:25.325
<00:26:22.745>of <00:26:22.945>requests <00:26:23.145>took <00:26:23.344>the <00:26:23.544>same <00:26:23.745>or <00:26:23.945>longer <00:26:24.145>time.

428
00:26:28.905 --> 00:26:33.035
<00:26:28.905>Also, <00:26:29.080>make <00:26:29.254>sure <00:26:29.430>that <00:26:29.605>your <00:26:29.780>load <00:26:29.955>is <00:26:30.129>even, <00:26:30.305>uh,

429
00:26:33.035 --> 00:26:35.915
<00:26:33.035>because <00:26:33.210>your <00:26:33.385>workload <00:26:33.560>might <00:26:33.735>be <00:26:33.910>skewed <00:26:34.085>when <00:26:34.260>a <00:26:34.435>single

430
00:26:35.915 --> 00:26:38.915
<00:26:35.915>partition <00:26:36.148>gets <00:26:36.381>more <00:26:36.615>throughput <00:26:36.848>than <00:26:37.081>other <00:26:37.315>partitions.

431
00:26:43.145 --> 00:26:47.205
<00:26:43.145>Um, <00:26:43.272>also <00:26:43.399>if <00:26:43.526>you <00:26:43.654>have <00:26:43.781>a <00:26:43.908>chance <00:26:44.035>to <00:26:44.163>use <00:26:44.290>real <00:26:44.417>data, <00:26:44.545>uh,

432
00:26:47.635 --> 00:26:49.165
<00:26:47.635>it's <00:26:48.323>usually <00:26:49.012>good.

433
00:26:49.785 --> 00:26:52.045
<00:26:49.785>So <00:26:50.018>test <00:26:50.251>with <00:26:50.485>real <00:26:50.718>data <00:26:50.951>whenever <00:26:51.185>possible.

434
00:26:58.775 --> 00:27:03.655
<00:26:58.775>So <00:27:00.175>you

435
00:27:03.655 --> 00:27:05.975
<00:27:03.655>need <00:27:03.810>to <00:27:03.966>know <00:27:04.121>the <00:27:04.277>capacity <00:27:04.432>of <00:27:04.588>your <00:27:04.743>system, <00:27:04.899>its <00:27:05.055>limits

436
00:27:06.115 --> 00:27:07.815
<00:27:06.115>and <00:27:06.395>how <00:27:06.675>much <00:27:06.955>headroom <00:27:07.235>you <00:27:07.515>have.

437
00:27:08.115 --> 00:27:10.135
<00:27:08.115>And <00:27:08.270>for <00:27:08.426>that <00:27:08.581>you <00:27:08.737>need <00:27:08.892>to <00:27:09.048>know <00:27:09.203>your <00:27:09.359>use <00:27:09.515>cases.

438
00:27:11.595 --> 00:27:14.015
<00:27:11.595>Uh, <00:27:11.770>so <00:27:11.945>like <00:27:12.120>how <00:27:12.295>much <00:27:12.470>traffic <00:27:12.645>do <00:27:12.820>you <00:27:12.995>expect?

439
00:27:14.085 --> 00:27:16.855
<00:27:14.085>What <00:27:14.240>is <00:27:14.396>your <00:27:14.551>spike <00:27:14.707>load, <00:27:14.862>what <00:27:15.018>is <00:27:15.173>your <00:27:15.329>consistent <00:27:15.485>load?

440
00:27:18.035 --> 00:27:22.205
<00:27:18.035>Uh, <00:27:18.235>but <00:27:18.435>then <00:27:18.635>with <00:27:18.835>monitoring <00:27:19.035>visibil, <00:27:19.235>visibil <00:27:19.435>observability,

441
00:27:22.345 --> 00:27:25.285
<00:27:22.345>you <00:27:22.545>can <00:27:22.745>download <00:27:22.945>testing <00:27:23.145>to <00:27:23.345>see <00:27:23.545>the <00:27:23.745>limits.

442
00:27:26.805 --> 00:27:29.185
<00:27:26.805>Uh, <00:27:27.271>so <00:27:27.738>the <00:27:28.205>monitoring

443
00:27:29.245 --> 00:27:33.105
<00:27:29.245>and <00:27:29.524>observability <00:27:29.804>of <00:27:30.084>your <00:27:30.364>Kafka <00:27:30.645>cluster

444
00:27:34.445 --> 00:27:38.985
<00:27:34.445>is <00:27:34.620>very, <00:27:34.794>very <00:27:34.970>important <00:27:35.145>as, <00:27:35.320>uh, <00:27:35.495>monitoring <00:27:35.669>helps, <00:27:35.845>uh,

445
00:27:38.985 --> 00:27:40.105
<00:27:38.985>provide <00:27:39.993>assurances

446
00:27:40.105 --> 00:27:42.505
<00:27:40.105>that <00:27:40.338>all <00:27:40.571>your <00:27:40.805>services <00:27:41.038>are <00:27:41.271>working <00:27:41.505>correctly

447
00:27:43.755 --> 00:27:46.975
<00:27:43.755>and <00:27:43.895>it <00:27:44.035>would <00:27:44.175>help <00:27:44.315>if <00:27:44.455>you <00:27:44.595>had <00:27:44.735>observability <00:27:44.875>in <00:27:45.015>your <00:27:45.155>system.

448
00:27:48.115 --> 00:27:51.255
<00:27:48.115>Uh, <00:27:48.315>without <00:27:48.515>monitoring <00:27:48.715>you <00:27:48.915>are <00:27:49.115>basically <00:27:49.315>driving <00:27:49.515>blind

449
00:27:53.105 --> 00:27:56.365
<00:27:53.105>and <00:27:53.245>out <00:27:53.385>of <00:27:53.525>the <00:27:53.665>box <00:27:53.805>Kafka <00:27:53.945>exposes <00:27:54.085>its <00:27:54.225>metrics <00:27:54.365>via <00:27:54.505>JMX

450
00:27:57.595 --> 00:28:01.615
<00:27:57.595>and <00:27:57.750>any <00:27:57.906>monitoring <00:27:58.061>tool <00:27:58.217>with <00:27:58.372>JMX <00:27:58.528>support <00:27:58.683>should <00:27:58.839>be <00:27:58.995>able

451
00:28:01.615 --> 00:28:05.695
<00:28:01.615>to <00:28:01.965>monitor, <00:28:02.315>uh, <00:28:02.665>Apache <00:28:03.015>Kafka.

452
00:28:08.295 --> 00:28:10.585
<00:28:08.295>And <00:28:08.995>uh, <00:28:09.695>yeah.

453
00:28:10.585 --> 00:28:12.105
<00:28:10.585>So <00:28:10.813>as <00:28:11.041>I <00:28:11.269>mentioned, <00:28:11.497>it <00:28:11.725>is <00:28:11.953>essential

454
00:28:12.105 --> 00:28:14.825
<00:28:12.105>to <00:28:12.338>collect <00:28:12.571>JMX <00:28:12.805>metrics <00:28:13.038>from <00:28:13.271>Kafka <00:28:13.505>brokers

455
00:28:16.725 --> 00:28:20.345
<00:28:16.725>and <00:28:16.899>beside <00:28:17.074>the <00:28:17.250>broker <00:28:17.425>metrics, <00:28:17.600>it <00:28:17.774>is <00:28:17.949>also <00:28:18.125>very,

456
00:28:20.345 --> 00:28:22.385
<00:28:20.345>very <00:28:20.625>important <00:28:20.905>to <00:28:21.185>measure <00:28:21.465>the <00:28:21.745>consumer

457
00:28:22.605 --> 00:28:27.025
<00:28:22.605>and <00:28:23.071>producer, <00:28:23.538>which <00:28:24.005>expose

458
00:28:27.185 --> 00:28:28.265
<00:28:27.185>JMX <00:28:27.509>metrics <00:28:27.833>as <00:28:28.157>well.

459
00:28:29.205 --> 00:28:32.425
<00:28:29.205>Uh, <00:28:29.360>moreover <00:28:29.516>it <00:28:29.671>would <00:28:29.827>help <00:28:29.982>if <00:28:30.138>you <00:28:30.293>collect <00:28:30.449>system <00:28:30.605>metrics,

460
00:28:33.645 --> 00:28:37.305
<00:28:33.645>uh, <00:28:33.800>make <00:28:33.956>sure <00:28:34.111>that <00:28:34.267>you <00:28:34.422>track <00:28:34.578>uh, <00:28:34.733>things <00:28:34.889>like, <00:28:35.045>uh,

461
00:28:37.325 --> 00:28:40.385
<00:28:37.325>memory <00:28:37.675>usage <00:28:38.025>JVM <00:28:38.375>hip <00:28:38.725>sites

462
00:28:42.885 --> 00:28:46.665
<00:28:42.885>and <00:28:43.118>like <00:28:43.351>things <00:28:43.585>like <00:28:43.818>swap <00:28:44.051>usage <00:28:44.285>network,

463
00:28:47.455 --> 00:28:51.305
<00:28:47.455>keep <00:28:47.688>an <00:28:47.921>eye <00:28:48.155>on <00:28:48.388>dropped <00:28:48.621>packets, <00:28:48.855>uh,

464
00:28:52.015 --> 00:28:53.705
<00:28:52.015>disk <00:28:52.481>IO <00:28:52.948>disk <00:28:53.415>usage.

465
00:28:54.535 --> 00:28:57.745
<00:28:54.535>Make <00:28:54.735>sure <00:28:54.935>you <00:28:55.135>always <00:28:55.335>have <00:28:55.535>enough <00:28:55.735>free <00:28:55.935>disc

466
00:28:57.755 --> 00:28:58.785
<00:28:57.755>space <00:28:58.064>for <00:28:58.373>new <00:28:58.682>data.

467
00:29:00.085 --> 00:29:04.225
<00:29:00.085>Uh, <00:29:00.260>your <00:29:00.435>disc <00:29:00.610>space <00:29:00.785>threshold <00:29:00.960>should <00:29:01.135>be <00:29:01.310>around <00:29:01.485>70

468
00:29:04.605 --> 00:29:07.265
<00:29:04.605>or <00:29:04.838>even <00:29:05.071>60% <00:29:05.305>of <00:29:05.538>the <00:29:05.771>disc <00:29:06.005>space.

469
00:29:10.005 --> 00:29:12.865
<00:29:10.005>And <00:29:10.145>here <00:29:10.285>on <00:29:10.425>these <00:29:10.565>slides, <00:29:10.705>there <00:29:10.845>are <00:29:10.985>two <00:29:11.125>examples <00:29:11.265>of, <00:29:11.405>uh,

470
00:29:13.095 --> 00:29:16.065
<00:29:13.095>popular <00:29:13.795>dashboard <00:29:14.495>tools.

471
00:29:16.325 --> 00:29:17.825
<00:29:16.325>One <00:29:16.517>of <00:29:16.710>them <00:29:16.903>is <00:29:17.096>Grafana <00:29:17.289>on <00:29:17.482>the <00:29:17.675>left

472
00:29:18.045 --> 00:29:20.105
<00:29:18.045>and <00:29:18.278>content <00:29:18.511>control <00:29:18.745>center <00:29:18.978>on <00:29:19.211>the <00:29:19.445>right.

473
00:29:28.375 --> 00:29:29.375
So,

474
00:29:30.465 --> 00:29:31.245
<00:29:30.465>Uh, <00:29:31.167>sorry,

475
00:29:37.655 --> 00:29:39.475
<00:29:37.655>Uh, <00:29:38.004>there <00:29:38.355>are <00:29:38.705>specialized <00:29:39.055>tools

476
00:29:39.535 --> 00:29:41.195
<00:29:39.535>to <00:29:40.001>test <00:29:40.468>Kafka's <00:29:40.935>performance.

477
00:29:41.505 --> 00:29:44.485
<00:29:41.505>Uh, <00:29:41.660>but <00:29:41.816>out <00:29:41.971>of <00:29:42.127>the <00:29:42.282>box <00:29:42.438>any <00:29:42.593>Kafka <00:29:42.749>installation <00:29:42.905>comes

478
00:29:44.555 --> 00:29:47.685
<00:29:44.555>with <00:29:44.730>command <00:29:44.905>line <00:29:45.080>tools <00:29:45.255>designed <00:29:45.430>to <00:29:45.605>do <00:29:45.780>just <00:29:45.955>that.

479
00:29:48.825 --> 00:29:51.565
<00:29:48.825>And <00:29:49.000>these <00:29:49.175>command <00:29:49.350>line <00:29:49.525>tools <00:29:49.700>are <00:29:49.875>Kafka <00:29:50.050>producer <00:29:50.225>performance

480
00:29:51.565 --> 00:29:53.525
<00:29:51.565>test <00:29:51.845>and <00:29:52.125>Kafka <00:29:52.405>consumer <00:29:52.685>performance <00:29:52.965>test.

481
00:29:53.905 --> 00:29:57.725
<00:29:53.905>And <00:29:54.105>you <00:29:54.305>can <00:29:54.504>run <00:29:54.705>performance <00:29:54.905>test <00:29:55.105>script, <00:29:55.305>uh,

482
00:29:57.725 --> 00:29:59.565
<00:29:57.725>with <00:29:58.191>different <00:29:58.658>configuration <00:29:59.125>settings.

483
00:30:01.225 --> 00:30:06.125
<00:30:01.225>And, <00:30:01.399>uh, <00:30:01.574>here <00:30:01.750>are <00:30:01.925>two <00:30:02.100>examples <00:30:02.274>of, <00:30:02.449>uh, <00:30:02.625>one

484
00:30:06.125 --> 00:30:08.045
<00:30:06.125>of <00:30:06.325>them <00:30:06.525>is <00:30:06.725>for <00:30:06.925>Kafka <00:30:07.125>producer <00:30:07.325>performance <00:30:07.525>test

485
00:30:08.145 --> 00:30:10.885
<00:30:08.145>and <00:30:08.320>another <00:30:08.495>one <00:30:08.670>is <00:30:08.845>for <00:30:09.020>Kafka <00:30:09.195>consumer <00:30:09.370>performance <00:30:09.545>test.

486
00:30:12.255 --> 00:30:17.235
<00:30:12.255>And <00:30:12.410>in <00:30:12.566>the <00:30:12.721>following <00:30:12.877>example, <00:30:13.032>we <00:30:13.188>will <00:30:13.343>produce <00:30:13.499>5,000 <00:30:13.655>messages

487
00:30:17.265 --> 00:30:19.235
<00:30:17.265>with <00:30:17.498>the <00:30:17.731>size <00:30:17.965>of <00:30:18.198>two <00:30:18.431>kilobytes <00:30:18.665>each.

488
00:30:20.935 --> 00:30:23.055
<00:30:20.935>And, <00:30:21.635>uh, <00:30:22.335>the

489
00:30:23.055 --> 00:30:25.495
<00:30:23.055>above <00:30:23.335>common <00:30:23.615>example <00:30:23.895>will <00:30:24.175>create <00:30:24.455>randomly

490
00:30:25.495 --> 00:30:26.695
<00:30:25.495>generated <00:30:26.034>by <00:30:26.574>record.

491
00:30:26.875 --> 00:30:28.375
<00:30:26.875>So <00:30:27.325>the <00:30:27.775>specified <00:30:28.225>size

492
00:30:30.365 --> 00:30:32.825
<00:30:30.365>and <00:30:30.565>the <00:30:30.765>minus <00:30:30.965>one <00:30:31.165>value <00:30:31.365>for <00:30:31.565>throughput <00:30:31.765>means

493
00:30:32.825 --> 00:30:35.505
<00:30:32.825>that <00:30:33.025>messages <00:30:33.225>are <00:30:33.425>produced <00:30:33.625>as <00:30:33.825>quickly <00:30:34.025>as <00:30:34.225>possible

494
00:30:35.775 --> 00:30:37.585
<00:30:35.775>with <00:30:36.241>no <00:30:36.708>ING <00:30:37.175>limit.

495
00:30:40.125 --> 00:30:44.405
<00:30:40.125>Um, <00:30:40.325>another <00:30:40.525>option <00:30:40.725>for <00:30:40.925>producing <00:30:41.125>test <00:30:41.325>data <00:30:41.525>is

496
00:30:44.405 --> 00:30:46.045
<00:30:44.405>to <00:30:44.638>read <00:30:44.871>it <00:30:45.105>from <00:30:45.338>a <00:30:45.571>file <00:30:45.805>randomly.

497
00:30:48.355 --> 00:30:52.135
<00:30:48.355>So <00:30:48.588>in <00:30:48.821>this <00:30:49.055>example, <00:30:49.288>uh, <00:30:49.521>333 <00:30:49.755>messages

498
00:30:52.675 --> 00:30:56.775
<00:30:52.675>are <00:30:52.830>produced <00:30:52.986>per <00:30:53.141>second <00:30:53.297>on <00:30:53.452>average <00:30:53.608>with <00:30:53.763>an <00:30:53.919>average <00:30:54.075>latency

499
00:30:56.775 --> 00:31:00.295
<00:30:56.775>of <00:30:57.125>approximately, <00:30:57.475>um, <00:30:57.825>7,500 <00:30:58.175>milliseconds

500
00:31:00.715 --> 00:31:04.335
<00:31:00.715>and <00:31:00.995>maximum <00:31:01.274>latency <00:31:01.554>of <00:31:01.834>14,500 <00:31:02.115>milliseconds.

501
00:31:05.915 --> 00:31:08.615
<00:31:05.915>And <00:31:06.264>the <00:31:06.615>99th <00:31:06.965>percentile <00:31:07.315>latency

502
00:31:08.755 --> 00:31:13.695
<00:31:08.755>of <00:31:08.930>13,400 <00:31:09.105>milliseconds <00:31:09.280>means <00:31:09.455>that <00:31:09.630>for <00:31:09.805>95% <00:31:09.980>of <00:31:10.155>messages,

503
00:31:13.755 --> 00:31:17.815
<00:31:13.755>it <00:31:13.988>took <00:31:14.221>less <00:31:14.455>than <00:31:14.688>13,400 <00:31:14.921>milliseconds <00:31:15.155>between

504
00:31:18.365 --> 00:31:19.455
<00:31:18.365>when <00:31:18.692>they <00:31:19.019>were <00:31:19.346>produced

505
00:31:19.555 --> 00:31:23.645
<00:31:19.555>and <00:31:19.755>when <00:31:19.955>they <00:31:20.155>were <00:31:20.355>written <00:31:20.555>on <00:31:20.755>all <00:31:20.955>brokers.

506
00:31:25.885 --> 00:31:30.465
<00:31:25.885>In <00:31:26.085>the <00:31:26.285>same <00:31:26.485>way, <00:31:26.685>the <00:31:26.885>14,500 <00:31:27.085>milliseconds <00:31:27.285>value

507
00:31:30.565 --> 00:31:34.585
<00:31:30.565>for <00:31:30.915>the <00:31:31.265>99.9 <00:31:31.615>percentile <00:31:31.965>means

508
00:31:34.585 --> 00:31:38.225
<00:31:34.585>that <00:31:34.935>one <00:31:35.285>in <00:31:35.635>thousand <00:31:35.985>messages

509
00:31:39.255 --> 00:31:41.305
<00:31:39.255>experienced <00:31:39.535>a <00:31:39.815>delay <00:31:40.095>of <00:31:40.375>more <00:31:40.655>than

510
00:31:42.025 --> 00:31:46.455
<00:31:42.025>14,000 <00:31:42.258>milliseconds <00:31:42.491>from <00:31:42.725>when <00:31:42.958>it <00:31:43.191>was <00:31:43.425>produced

511
00:31:46.505 --> 00:31:48.935
<00:31:46.505>until <00:31:46.680>it <00:31:46.855>was <00:31:47.030>published <00:31:47.205>and <00:31:47.380>committed <00:31:47.555>on <00:31:47.730>all <00:31:47.905>brokers.

512
00:31:51.485 --> 00:31:53.935
<00:31:51.485>Also, <00:31:51.718>there <00:31:51.951>is <00:31:52.185>a <00:31:52.418>print <00:31:52.651>metrics <00:31:52.885>parameter,

513
00:31:53.935 --> 00:31:58.015
<00:31:53.935>which <00:31:54.090>prints <00:31:54.246>out <00:31:54.401>the <00:31:54.557>producer <00:31:54.712>MX <00:31:54.868>metrics <00:31:55.023>at <00:31:55.179>the <00:31:55.335>end,

514
00:31:59.905 --> 00:32:03.125
<00:31:59.905>uh, <00:32:00.032>which <00:32:00.159>is <00:32:00.286>very <00:32:00.414>useful <00:32:00.541>if <00:32:00.668>you <00:32:00.795>want <00:32:00.923>to <00:32:01.050>grab <00:32:01.177>your <00:32:01.305>producer

515
00:32:03.225 --> 00:32:07.685
<00:32:03.225>or <00:32:03.399>consumer <00:32:03.574>JMX <00:32:03.750>metrics <00:32:03.925>and <00:32:04.100>to <00:32:04.274>look <00:32:04.449>at <00:32:04.625>them

516
00:32:07.895 --> 00:32:09.365
<00:32:07.895>after <00:32:08.225>you <00:32:08.556>run <00:32:08.887>the <00:32:09.218>test.

517
00:32:13.435 --> 00:32:17.415
<00:32:13.435>Uh, <00:32:13.610>kaka, <00:32:13.784>uh, <00:32:13.960>yeah, <00:32:14.135>so <00:32:14.310>Apache <00:32:14.485>Kafka <00:32:14.659>has, <00:32:14.835>uh,

518
00:32:17.925 --> 00:32:22.125
<00:32:17.925>builtin <00:32:18.125>test, <00:32:18.325>uh, <00:32:18.524>sorry, <00:32:18.725>uh, <00:32:18.925>builtin <00:32:19.125>test <00:32:19.325>framework

519
00:32:22.695 --> 00:32:27.405
<00:32:22.695>named, <00:32:22.850>uh, <00:32:23.006>that <00:32:23.161>can <00:32:23.317>be <00:32:23.472>used <00:32:23.628>to <00:32:23.783>perform <00:32:23.939>fault <00:32:24.095>injection

520
00:32:27.745 --> 00:32:29.205
<00:32:27.745>or <00:32:28.183>to <00:32:28.620>execute <00:32:29.059>workloads.

521
00:32:29.305 --> 00:32:33.045
<00:32:29.305>Uh, <00:32:29.480>for <00:32:29.655>a <00:32:29.830>test <00:32:30.005>system, <00:32:30.180>it <00:32:30.355>is <00:32:30.530>open <00:32:30.705>source

522
00:32:33.145 --> 00:32:35.565
<00:32:33.145>and <00:32:33.345>part <00:32:33.545>of <00:32:33.745>your <00:32:33.945>Kafka <00:32:34.145>installation <00:32:34.345>as <00:32:34.545>well.

523
00:32:36.265 --> 00:32:40.845
<00:32:36.265>And <00:32:36.440>truck <00:32:36.615>door <00:32:36.790>can <00:32:36.965>run <00:32:37.140>benchmarks <00:32:37.315>and <00:32:37.490>other <00:32:37.665>workloads,

524
00:32:41.985 --> 00:32:46.165
<00:32:41.985>and <00:32:42.218>also <00:32:42.451>drug <00:32:42.685>dock <00:32:42.918>can <00:32:43.151>inject <00:32:43.385>faults

525
00:32:47.425 --> 00:32:49.725
<00:32:47.425>in <00:32:47.658>your <00:32:47.891>system <00:32:48.125>to <00:32:48.358>stress <00:32:48.591>test <00:32:48.825>it.

526
00:32:51.505 --> 00:32:56.365
<00:32:51.505>Uh, <00:32:51.855>so <00:32:52.205>Truckdo <00:32:52.555>has, <00:32:52.905>uh,

527
00:32:56.425 --> 00:32:59.645
<00:32:56.425>single <00:32:56.705>coordinator <00:32:56.985>process, <00:32:57.264>which <00:32:57.544>manages <00:32:57.825>multiple

528
00:33:00.295 --> 00:33:02.045
<00:33:00.295>agent <00:33:00.761>processes <00:33:01.228>and <00:33:01.695>tasks.

529
00:33:02.705 --> 00:33:06.005
<00:33:02.705>And <00:33:02.821>the <00:33:02.938>task <00:33:03.054>is <00:33:03.171>anything <00:33:03.288>we <00:33:03.405>might <00:33:03.521>want <00:33:03.638>to <00:33:03.754>do <00:33:03.871>on <00:33:03.988>a <00:33:04.105>cluster

530
00:33:06.915 --> 00:33:10.045
<00:33:06.915>such <00:33:07.090>as <00:33:07.264>running <00:33:07.440>a <00:33:07.615>benchmark <00:33:07.790>or <00:33:07.965>injecting <00:33:08.139>a <00:33:08.315>fault

531
00:33:10.225 --> 00:33:11.325
<00:33:10.225>or <00:33:10.554>running <00:33:10.885>a <00:33:11.215>workload.

532
00:33:12.745 --> 00:33:16.965
<00:33:12.745>Uh, <00:33:12.919>this <00:33:13.094>task <00:33:13.270>specification <00:33:13.445>is <00:33:13.620>usually <00:33:13.794>written <00:33:13.969>as <00:33:14.145>Js

533
00:33:17.345 --> 00:33:19.365
<00:33:17.345>and <00:33:17.625>example <00:33:17.905>of <00:33:18.185>this <00:33:18.465>JSON <00:33:18.745>configuration

534
00:33:19.625 --> 00:33:20.725
<00:33:19.625>can <00:33:19.823>be <00:33:20.021>seen <00:33:20.219>on <00:33:20.417>the <00:33:20.615>slide.

535
00:33:22.665 --> 00:33:27.045
<00:33:22.665>And, <00:33:22.865>uh, <00:33:23.065>the <00:33:23.264>source, <00:33:23.465>uh, <00:33:23.665>repository <00:33:23.865>of <00:33:24.065>Apache,

536
00:33:27.045 --> 00:33:30.765
<00:33:27.045>of <00:33:27.220>Apache <00:33:27.395>Kafka <00:33:27.570>includes <00:33:27.745>an <00:33:27.920>extensive, <00:33:28.095>uh, <00:33:28.270>test <00:33:28.445>sheet

537
00:33:30.785 --> 00:33:32.885
<00:33:30.785>for <00:33:31.065>system <00:33:31.345>integration <00:33:31.625>and <00:33:31.905>performance <00:33:32.185>tests.

538
00:33:34.065 --> 00:33:35.805
<00:33:34.065>And <00:33:34.265>there <00:33:34.465>is <00:33:34.665>a <00:33:34.865>link <00:33:35.065>on <00:33:35.265>the <00:33:35.465>slide.

539
00:33:40.925 --> 00:33:45.815
<00:33:40.925>Another <00:33:41.100>popular <00:33:41.274>tool, <00:33:41.450>uh, <00:33:41.625>which <00:33:41.800>Alterna <00:33:41.975>can <00:33:42.149>be <00:33:42.325>used

540
00:33:45.915 --> 00:33:50.495
<00:33:45.915>for <00:33:46.090>specialized <00:33:46.264>for <00:33:46.440>performance <00:33:46.615>testing <00:33:46.790>is <00:33:46.965>Apache <00:33:47.139>J <00:33:47.315>meter

541
00:33:50.715 --> 00:33:55.655
<00:33:50.715>and <00:33:50.915>pepper <00:33:51.115>box, <00:33:51.314>uh, <00:33:51.514>plugin <00:33:51.715>for <00:33:51.915>Kafka, <00:33:52.115>uh,

542
00:33:55.805 --> 00:33:58.535
<00:33:55.805>load <00:33:56.505>generation, <00:33:57.205>uh,

543
00:33:58.755 --> 00:34:00.975
<00:33:58.755>to <00:33:59.035>load <00:33:59.315>test <00:33:59.595>your <00:33:59.875>Kafka <00:34:00.155>installation.

544
00:34:01.875 --> 00:34:05.415
<00:34:01.875>So <00:34:02.050>Apache <00:34:02.225>JMeter <00:34:02.400>is <00:34:02.575>a <00:34:02.750>Java <00:34:02.925>desktop <00:34:03.100>application <00:34:03.275>designed

545
00:34:06.835 --> 00:34:08.535
<00:34:06.835>to <00:34:07.115>load, <00:34:07.395>test <00:34:07.675>and <00:34:07.955>measure <00:34:08.235>performance.

546
00:34:08.675 --> 00:34:11.895
<00:34:08.675>And <00:34:08.850>it <00:34:09.025>was <00:34:09.200>initially <00:34:09.375>designed <00:34:09.550>for <00:34:09.725>testing <00:34:09.900>web <00:34:10.075>applications,

547
00:34:12.395 --> 00:34:16.655
<00:34:12.395>uh, <00:34:12.570>but <00:34:12.745>has <00:34:12.920>since <00:34:13.094>expanded <00:34:13.270>to <00:34:13.445>other <00:34:13.620>test <00:34:13.795>functions.

548
00:34:18.415 --> 00:34:21.595
<00:34:18.415>And <00:34:18.614>this <00:34:18.815>way, <00:34:19.014>uh, <00:34:19.215>using <00:34:19.415>Apache <00:34:19.614>J <00:34:19.815>meter

549
00:34:21.615 --> 00:34:26.355
<00:34:21.614>and <00:34:21.742>pepper <00:34:21.869>box, <00:34:21.996>you <00:34:22.124>will <00:34:22.251>be <00:34:22.378>able <00:34:22.505>to <00:34:22.633>conduct <00:34:22.760>the <00:34:22.887>load, <00:34:23.014>uh,

550
00:34:26.455 --> 00:34:31.075
<00:34:26.455>having <00:34:26.921>full, <00:34:27.388>uh, <00:34:27.855>sorry

551
00:34:31.495 --> 00:34:35.595
<00:34:31.495>to <00:34:31.634>conduct, <00:34:31.775>uh, <00:34:31.915>tests, <00:34:32.054>uh, <00:34:32.194>where <00:34:32.335>you <00:34:32.475>can <00:34:32.614>have <00:34:32.755>full <00:34:32.895>control

552
00:34:35.655 --> 00:34:38.995
<00:34:35.655>of <00:34:35.888>threats, <00:34:36.121>ramp <00:34:36.355>up <00:34:36.588>time, <00:34:36.821>message <00:34:37.055>size,

553
00:34:39.465 --> 00:34:40.875
<00:34:39.465>content <00:34:39.888>and <00:34:40.311>so <00:34:40.734>on.

554
00:34:42.055 --> 00:34:45.955
<00:34:42.054>And <00:34:42.194>also <00:34:42.335>you <00:34:42.475>will <00:34:42.614>be <00:34:42.754>able <00:34:42.895>to <00:34:43.034>generate <00:34:43.174>STML <00:34:43.315>uh, <00:34:43.455>dashboard,

555
00:34:45.975 --> 00:34:50.235
<00:34:45.975>having <00:34:46.130>tables <00:34:46.286>and <00:34:46.441>charts, <00:34:46.597>uh, <00:34:46.752>with <00:34:46.908>all <00:34:47.063>the <00:34:47.219>useful <00:34:47.375>metrics.

556
00:34:56.445 --> 00:34:58.385
<00:34:56.445>So <00:34:57.845>performance

557
00:34:58.685 --> 00:35:01.705
<00:34:58.685>and <00:34:59.035>performance <00:34:59.384>measurement <00:34:59.735>are <00:35:00.085>multidimensional

558
00:35:02.565 --> 00:35:04.665
<00:35:02.565>and <00:35:02.845>they <00:35:03.125>depend <00:35:03.405>on <00:35:03.685>multiple <00:35:03.965>parameters,

559
00:35:04.785 --> 00:35:06.425
<00:35:04.785>settings, <00:35:05.484>and <00:35:06.185>conditions.

560
00:35:08.265 --> 00:35:09.445
<00:35:08.265>You <00:35:08.477>need <00:35:08.689>to <00:35:08.902>understand <00:35:09.114>the <00:35:09.327>system

561
00:35:09.465 --> 00:35:11.205
<00:35:09.465>and <00:35:09.665>to <00:35:09.865>be <00:35:10.065>able <00:35:10.265>to <00:35:10.465>explain <00:35:10.665>the <00:35:10.865>results,

562
00:35:11.905 --> 00:35:15.325
<00:35:11.905>and <00:35:12.105>it <00:35:12.305>would <00:35:12.505>help <00:35:12.705>if <00:35:12.905>you <00:35:13.105>approach <00:35:13.305>performance

563
00:35:15.725 --> 00:35:18.165
<00:35:15.725>measurements <00:35:17.125>mind.

564
00:35:20.415 --> 00:35:20.885
<00:35:20.415>Thank <00:35:20.838>you.