Kirill Kulikov - How to Benchmark Your Apache Kafka

#1about 4 minutes

A brief overview of Apache Kafka architecture

Key concepts like events, topics, partitions, replicas, producers, and consumers form the foundation of Kafka's distributed system.

#2about 2 minutes

Choosing between throughput and latency optimization

Understand your specific use case to decide whether to prioritize moving large volumes of data (throughput) or ensuring messages arrive quickly (latency).

#3about 5 minutes

Understanding and optimizing for low latency

Minimize end-to-end message delivery time by tuning producer parameters like linger.ms and acks, and consumer settings like fetch.min.bytes.

#4about 3 minutes

Using the sticky partitioner to improve batching

The sticky partitioner improves batching for messages without keys by sending them to a single partition until a batch is full, reducing latency compared to the old round-robin strategy.

#5about 6 minutes

How to configure Kafka for high throughput

Maximize the rate of data movement by increasing partition count, tuning producer batching with batch.size and linger.ms, and enabling compression.

#6about 5 minutes

Strategies and best practices for load testing

Employ stress, spike, and soak testing methods while following best practices like system warmup, multiple runs, and analyzing percentiles to get accurate performance results.

#7about 3 minutes

Monitoring Kafka clusters with JMX and system metrics

Gain observability into your cluster's health by collecting JMX metrics from brokers and clients, alongside essential system metrics like CPU, disk, and network usage.

#8about 3 minutes

Using Kafka's built-in performance testing scripts

Leverage the out-of-the-box kafka-producer-perf-test and kafka-consumer-perf-test command-line tools for basic performance benchmarking.

#9about 3 minutes

Advanced benchmarking with Trogdor and JMeter

Use advanced tools like Trogdor for fault injection and workload execution, or Apache JMeter with the Pepper-Box plugin for more complex and controlled load testing scenarios.

1 month ago

Senior Software Engineer