WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Nov 30, 2020

How to Benchmark Your Apache Kafka

Kirill Kulikov

Why does setting acks=1 sacrifice durability without actually improving end-to-end latency? Uncover the true configuration trade-offs needed to benchmark and optimize your Apache Kafka clusters.

Pause
Mute Enter Fullscreen
#1 about 2 min

Introduction to Apache Kafka benchmarking and performance analysis

Why benchmarking is critical for understanding cluster capacity and optimizing resource utilization.

#2 about 4 min

Reviewing core Apache Kafka architecture and distributed fundamentals

An overview of events, topics, partitions, consumer groups, and the append-only log.

#3 about 2 min

Understanding throughput and latency trade-offs for varied workloads

How workload requirements dictate the service goals when configuring an Apache Kafka cluster.

#4 about 2 min

Measuring end-to-end latency components across the message lifecycle

Breaking down processing time across producing, publishing, replication, and fetching stages.

#5 about 4 min

Tuning producer and consumer configurations for low latency

Adjusting batch size, linger times, acks, and fetch sizes to ensure rapid message delivery.

#6 about 3 min

Reducing end-to-end latency using the sticky partitioner strategy

How the sticky partition strategy improves batching efficiency for non-keyed records to decrease wait times.

#7 about 7 min

Scaling topic partitions and configurations to maximize throughput

Maximizing data transmission rates through partition scaling, batch optimization, and selection of compression algorithms.

#8 about 2 min

Defining objectives and methodologies for systematic load testing

Exploring stress, spike, and soak testing methodologies to evaluate system behavioral limits under heavy use.

#9 about 4 min

Best practices for environment setup and executing load tests

Ensuring accurate results through dedicated test environments, JVM warm-up phases, and interpreting percentile metrics.

#10 about 3 min

Tracking observability metrics to ensure healthy cluster performance

Monitoring hardware usage, system state, and structural JMX metrics to maintain overall cluster health during operation.

#11 about 3 min

Using native command-line tools for synthetic performance testing

Utilizing built-in producer and consumer CLI scripts to generate specific computational loads and measure throughput accurately.

#12 about 2 min

Executing workloads and injecting system faults with Trogdor

Leveraging Apache Kafka's built-in framework to run complex benchmarks and simulate resilient system failures.

#13 about 2 min

Conducting advanced load testing workflows with Apache JMeter

Using the Pepper-Box plugin to finely control request threads, message payload sizes, and construct detailed web dashboards.

Matching moments

6:21 min

Audience questions on brokers and monitoring tools

Developersteve · LIVE

2:23 min

Benchmarking StoteFlow against Kafka Streams

Hartmut Armbruster Hartmut Armbruster · World Congress 2026 Europe

3:06 min

Core concepts of Apache Kafka and topic topologies

Denis Washington +1 · World Congress 2021

1:39 min

Defining Apache Kafka and its primary real-time use cases

Lucia Cerchie · LIVE

1:52 min

Real-world Apache Kafka production use cases

Developersteve · LIVE

5:03 min

Introduction to Apache Kafka as architecture glue

Developersteve · LIVE

More from WeAreDevelopers LIVE

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail
5:46
5 days ago

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

WeAreDevelopers - Of Agents and Robots thumbnail
1:04:42
5 days ago

WeAreDevelopers - Of Agents and Robots

Chris Heilmann, Daniel Cranney, Marc Plogas

WeAreDevelopers LIVE - How DevRel Makes Tech More Human thumbnail
48:06
12 days ago

WeAreDevelopers LIVE - How DevRel Makes Tech More Human

Chris Heilmann, Daniel Cranney

Fake or News: Smart Glasses in Court, Medical Advice Watches & Mushroom Electonics - Jason Torres thumbnail
7:41
13 days ago

Fake or News: Smart Glasses in Court, Medical Advice Watches & Mushroom Electonics - Jason Torres

Chris Heilmann, Daniel Cranney

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
26 days ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Node and Package Security thumbnail
1:08:30
26 days ago

WeAreDevelopers LIVE - Node and Package Security

Chris Heilmann, Daniel Cranney, Zbyszek Tenerowicz

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Your Thread Pool Is Lying to You — Sizing Concurrency from Rate Limits and Latency, Not Guesswork

Ratul Ghosh, Sesha Chennupati

Ratul Ghosh
Sesha Chennupati
Open session

World Congress 2026 North America

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

How to generate business value through performance optimizations

Nikolai Sidiropulo

Software Engineer at Meta

Nikolai Sidiropulo
Open session

World Congress 2026 North America

KV Cache Is Not About Speed: It's About Surviving Inference Costs

David vonThenen

AI/ML Leader | Keynote Speaker | OSS Engineer & Developer Advocate | Agentic AI, Deep Learning, Production AI | Python, Go, C++

David vonThenen
Open session

World Congress 2026 North America

Replay-Safe Architecture: Building Event-Driven Systems That Can Recover With Confidence

Ishan Shah

PayPal, Software Engineer | Distributed Systems, AI, and Platform Engineering

Ishan Shah
Open session

World Congress 2026 North America

Engineering Moneyball: How We Benchmarked Google vs Meta

Jirka Bachel

Co-Founder & CEO

Jirka Bachel

Related videos

Tips, Techniques, and Common Pitfalls Debugging Kafka thumbnail
54:29
LIVE • over 2 years ago

Tips, Techniques, and Common Pitfalls Debugging Kafka

Developersteve

Let's Get Started With Apache Kafka® for Python Developers thumbnail
38:50
LIVE • over 3 years ago

Let's Get Started With Apache Kafka® for Python Developers

Lucia Cerchie

Kafka Streams Microservices thumbnail
45:48
World Congress 2021 • about 5 years ago

Kafka Streams Microservices

Denis Washington, Olli Salonen

What If We've Been Scaling Stream Processing Wrong All Along? thumbnail
31:06
World Congress 2026 Europe • about 2 months ago

What If We've Been Scaling Stream Processing Wrong All Along?

Hartmut Armbruster

Why and when should we consider Stream Processing frameworks in our solutions thumbnail
30:51
World Congress 2024 • about 2 years ago

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Data binning and understanding histograms thumbnail
30:35
Europe 2026 Virtual • 2 months ago

Data binning and understanding histograms

Michal Bojko

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.