Kirill Kulikov
Nov 30, 2020
How to Benchmark Your Apache Kafka
#1about 4 minutes
A brief overview of Apache Kafka architecture
Key concepts like events, topics, partitions, replicas, producers, and consumers form the foundation of Kafka's distributed system.
#2about 2 minutes
Choosing between throughput and latency optimization
Understand your specific use case to decide whether to prioritize moving large volumes of data (throughput) or ensuring messages arrive quickly (latency).
#3about 5 minutes
Understanding and optimizing for low latency
Minimize end-to-end message delivery time by tuning producer parameters like linger.ms and acks, and consumer settings like fetch.min.bytes.
#4about 3 minutes
Using the sticky partitioner to improve batching
The sticky partitioner improves batching for messages without keys by sending them to a single partition until a batch is full, reducing latency compared to the old round-robin strategy.
#5about 6 minutes
How to configure Kafka for high throughput
Maximize the rate of data movement by increasing partition count, tuning producer batching with batch.size and linger.ms, and enabling compression.
#6about 5 minutes
Strategies and best practices for load testing
Employ stress, spike, and soak testing methods while following best practices like system warmup, multiple runs, and analyzing percentiles to get accurate performance results.
#7about 3 minutes
Monitoring Kafka clusters with JMX and system metrics
Gain observability into your cluster's health by collecting JMX metrics from brokers and clients, alongside essential system metrics like CPU, disk, and network usage.
#8about 3 minutes
Using Kafka's built-in performance testing scripts
Leverage the out-of-the-box kafka-producer-perf-test and kafka-consumer-perf-test command-line tools for basic performance benchmarking.
#9about 3 minutes
Advanced benchmarking with Trogdor and JMeter
Use advanced tools like Trogdor for fault injection and workload execution, or Apache JMeter with the Pepper-Box plugin for more complex and controlled load testing scenarios.
Related jobs
Jobs that call for the skills explored in this talk.
1 month ago
Senior Software Engineer

Riverty
Berlin, Germany
Senior
1 month ago
Digital Product Manager - Automation Services (m/w/d)

WALTER GROUP
Intermediate
Senior
yesterday
Java / Kotlin Developer in einem Cloud-Native-Stack

PROSOZ Herten GmbH
Herten, Germany
Intermediate
Senior


