WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Nov 30, 2020

How to Benchmark Your Apache Kafka

Kirill Kulikov

Why does setting acks=1 sacrifice durability without actually improving end-to-end latency? Uncover the true configuration trade-offs needed to benchmark and optimize your Apache Kafka clusters.

Pause
Mute Enter Fullscreen
#1 about 2 min

Introduction to Apache Kafka benchmarking and performance analysis

Why benchmarking is critical for understanding cluster capacity and optimizing resource utilization.

#2 about 4 min

Reviewing core Apache Kafka architecture and distributed fundamentals

An overview of events, topics, partitions, consumer groups, and the append-only log.

#3 about 2 min

Understanding throughput and latency trade-offs for varied workloads

How workload requirements dictate the service goals when configuring an Apache Kafka cluster.

#4 about 2 min

Measuring end-to-end latency components across the message lifecycle

Breaking down processing time across producing, publishing, replication, and fetching stages.

#5 about 4 min

Tuning producer and consumer configurations for low latency

Adjusting batch size, linger times, acks, and fetch sizes to ensure rapid message delivery.

#6 about 3 min

Reducing end-to-end latency using the sticky partitioner strategy

How the sticky partition strategy improves batching efficiency for non-keyed records to decrease wait times.

#7 about 7 min

Scaling topic partitions and configurations to maximize throughput

Maximizing data transmission rates through partition scaling, batch optimization, and selection of compression algorithms.

#8 about 2 min

Defining objectives and methodologies for systematic load testing

Exploring stress, spike, and soak testing methodologies to evaluate system behavioral limits under heavy use.

#9 about 4 min

Best practices for environment setup and executing load tests

Ensuring accurate results through dedicated test environments, JVM warm-up phases, and interpreting percentile metrics.

#10 about 3 min

Tracking observability metrics to ensure healthy cluster performance

Monitoring hardware usage, system state, and structural JMX metrics to maintain overall cluster health during operation.

#11 about 3 min

Using native command-line tools for synthetic performance testing

Utilizing built-in producer and consumer CLI scripts to generate specific computational loads and measure throughput accurately.

#12 about 2 min

Executing workloads and injecting system faults with Trogdor

Leveraging Apache Kafka's built-in framework to run complex benchmarks and simulate resilient system failures.

#13 about 2 min

Conducting advanced load testing workflows with Apache JMeter

Using the Pepper-Box plugin to finely control request threads, message payload sizes, and construct detailed web dashboards.

Matching moments

6:21 min

Audience questions on brokers and monitoring tools

Developersteve · LIVE

2:23 min

Benchmarking StoteFlow against Kafka Streams

Hartmut Armbruster Hartmut Armbruster · WWC Europe 2026

3:06 min

Core concepts of Apache Kafka and topic topologies

Denis Washington +1 · WWC 2021

1:39 min

Defining Apache Kafka and its primary real-time use cases

Lucia Cerchie · LIVE

1:52 min

Real-world Apache Kafka production use cases

Developersteve · LIVE

5:03 min

Introduction to Apache Kafka as architecture glue

Developersteve · LIVE

More from WeAreDevelopers LIVE

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
5 days ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Node and Package Security thumbnail
1:08:30
5 days ago

WeAreDevelopers LIVE - Node and Package Security

Chris Heilmann, Daniel Cranney, Zbyszek Tenerowicz

WeAreDevelopers LIVE - Streaming HTML thumbnail
1:10:26
12 days ago

WeAreDevelopers LIVE - Streaming HTML

Chris Heilmann, Daniel Cranney, Jad Joubran

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe thumbnail
1:02:34
19 days ago

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe

Chris Heilmann, Daniel Cranney, Justin Halsall, Werner Jeschko, Nicole Jeschko, Mirco Tomić

WeAreDevelopers LIVE - Web Performance in an AI World thumbnail
1:07:59
26 days ago

WeAreDevelopers LIVE - Web Performance in an AI World

Chris Heilmann, Daniel Cranney

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko thumbnail
5:55
about 2 months ago

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

World Congress 2026 North America

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

How to generate business value through performance optimizations

Nikolai Sidiropulo

Software Engineer at Meta

Nikolai Sidiropulo
Open session

World Congress 2026 North America

KV Cache Is Not About Speed: It's About Surviving Inference Costs

David vonThenen

AI/ML Leader | Keynote Speaker | OSS Engineer & Developer Advocate | Agentic AI, Deep Learning, Production AI | Python, Go, C++

David vonThenen
Open session

World Congress 2026 North America

The Autonomous Performance Agent: A Netflix Production Story

Rajat Shah

Staff Software Engineer, AI Platform, Netflix

Rajat Shah
Open session

World Congress 2026 North America

The $0.15/month Algo-Trader: Architecting High-Performance Serverless Trading Engines

Vishwak Thatikonda

Lead Full Stack Engineer at SDH Systems

Vishwak Thatikonda
Open session

World Congress 2026 North America

Run your agents in Kubernetes: Build once, deploy anywhere. But really?

Michal Salanci

Senior Systems Engineer at ESET Cybersecurity

Michal Salanci

Related videos

Tips, Techniques, and Common Pitfalls Debugging Kafka thumbnail
54:29
LIVE • over 2 years ago

Tips, Techniques, and Common Pitfalls Debugging Kafka

Developersteve

Let's Get Started With Apache Kafka® for Python Developers thumbnail
38:50
LIVE • over 3 years ago

Let's Get Started With Apache Kafka® for Python Developers

Lucia Cerchie

Kafka Streams Microservices thumbnail
45:48
WWC 2021 • about 5 years ago

Kafka Streams Microservices

Denis Washington, Olli Salonen

What If We've Been Scaling Stream Processing Wrong All Along? thumbnail
31:06
WWC Europe 2026 • about 1 month ago

What If We've Been Scaling Stream Processing Wrong All Along?

Hartmut Armbruster

Why and when should we consider Stream Processing frameworks in our solutions thumbnail
30:51
WWC 2024 • almost 2 years ago

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Data binning and understanding histograms thumbnail
30:35
Europe 2026 Virtual • about 1 month ago

Data binning and understanding histograms

Michal Bojko

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.