WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Nov 30, 2020

How to Benchmark Your Apache Kafka

Kirill Kulikov

Why does setting acks=1 sacrifice durability without actually improving end-to-end latency? Uncover the true configuration trade-offs needed to benchmark and optimize your Apache Kafka clusters.

Pause
Mute Enter Fullscreen
#1 about 2 min

Introduction to Apache Kafka benchmarking and performance analysis

Why benchmarking is critical for understanding cluster capacity and optimizing resource utilization.

#2 about 4 min

Reviewing core Apache Kafka architecture and distributed fundamentals

An overview of events, topics, partitions, consumer groups, and the append-only log.

#3 about 2 min

Understanding throughput and latency trade-offs for varied workloads

How workload requirements dictate the service goals when configuring an Apache Kafka cluster.

#4 about 2 min

Measuring end-to-end latency components across the message lifecycle

Breaking down processing time across producing, publishing, replication, and fetching stages.

#5 about 4 min

Tuning producer and consumer configurations for low latency

Adjusting batch size, linger times, acks, and fetch sizes to ensure rapid message delivery.

#6 about 3 min

Reducing end-to-end latency using the sticky partitioner strategy

How the sticky partition strategy improves batching efficiency for non-keyed records to decrease wait times.

#7 about 7 min

Scaling topic partitions and configurations to maximize throughput

Maximizing data transmission rates through partition scaling, batch optimization, and selection of compression algorithms.

#8 about 2 min

Defining objectives and methodologies for systematic load testing

Exploring stress, spike, and soak testing methodologies to evaluate system behavioral limits under heavy use.

#9 about 4 min

Best practices for environment setup and executing load tests

Ensuring accurate results through dedicated test environments, JVM warm-up phases, and interpreting percentile metrics.

#10 about 3 min

Tracking observability metrics to ensure healthy cluster performance

Monitoring hardware usage, system state, and structural JMX metrics to maintain overall cluster health during operation.

#11 about 3 min

Using native command-line tools for synthetic performance testing

Utilizing built-in producer and consumer CLI scripts to generate specific computational loads and measure throughput accurately.

#12 about 2 min

Executing workloads and injecting system faults with Trogdor

Leveraging Apache Kafka's built-in framework to run complex benchmarks and simulate resilient system failures.

#13 about 2 min

Conducting advanced load testing workflows with Apache JMeter

Using the Pepper-Box plugin to finely control request threads, message payload sizes, and construct detailed web dashboards.

Matching moments

6:21 min

Audience questions on brokers and monitoring tools

Developersteve · LIVE

2:23 min

Benchmarking StoteFlow against Kafka Streams

Hartmut Armbruster Hartmut Armbruster · World Congress 2026 Europe

3:06 min

Core concepts of Apache Kafka and topic topologies

Denis Washington +1 · World Congress 2021

1:39 min

Defining Apache Kafka and its primary real-time use cases

Lucia Cerchie · LIVE

1:52 min

Real-world Apache Kafka production use cases

Developersteve · LIVE

5:03 min

Introduction to Apache Kafka as architecture glue

Developersteve · LIVE

More from WeAreDevelopers LIVE

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude thumbnail
6:16
6 days ago

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude

Chris Heilmann, Daniel Cranney

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail
5:46
11 days ago

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

WeAreDevelopers - Of Agents and Robots thumbnail
1:04:42
11 days ago

WeAreDevelopers - Of Agents and Robots

Chris Heilmann, Daniel Cranney, Marc Plogas

WeAreDevelopers LIVE - How DevRel Makes Tech More Human thumbnail
48:06
18 days ago

WeAreDevelopers LIVE - How DevRel Makes Tech More Human

Chris Heilmann, Daniel Cranney

Fake or News: Smart Glasses in Court, Medical Advice Watches & Mushroom Electonics - Jason Torres thumbnail
7:41
19 days ago

Fake or News: Smart Glasses in Court, Medical Advice Watches & Mushroom Electonics - Jason Torres

Chris Heilmann, Daniel Cranney

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
about 1 month ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 25, 2026 · 09:00–09:30

Stage 3

Your Thread Pool Is Lying to You — Sizing Concurrency from Rate Limits and Latency, Not Guesswork

Ratul Ghosh, Sesha Chennupati

Ratul Ghosh
Sesha Chennupati
Open session

World Congress 2026 North America

September 24, 2026 · 16:50–17:20

Stage 9

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 3

Real-Time Data Platforms at Trillion-Event Scale

Diptamay Sanyal

Principal Engineer | Data, AI & Cybersecurity Platforms

Diptamay Sanyal
Open session

World Congress 2026 North America

September 24, 2026 · 14:50–15:20

Stage 9

How to generate business value through performance optimizations

Nikolai Sidiropulo

Software Engineer at Meta

Nikolai Sidiropulo
Open session

World Congress 2026 North America

September 24, 2026 · 15:30–16:00

Stage 1

KV Cache Is Not About Speed: It's About Surviving Inference Costs

David vonThenen

AI/ML Leader | Keynote Speaker | OSS Engineer & Developer Advocate | Agentic AI, Deep Learning, Production AI | Python, Go, C++

David vonThenen
Open session

World Congress 2026 North America

September 25, 2026 · 12:55–13:25

Stage 3

Replay-Safe Architecture: Building Event-Driven Systems That Can Recover With Confidence

Ishan Shah

PayPal, Software Engineer | Distributed Systems, AI, and Platform Engineering

Ishan Shah

Related videos

Tips, Techniques, and Common Pitfalls Debugging Kafka thumbnail
54:29
LIVE • over 2 years ago

Tips, Techniques, and Common Pitfalls Debugging Kafka

Developersteve

Let's Get Started With Apache Kafka® for Python Developers thumbnail
38:50
LIVE • over 3 years ago

Let's Get Started With Apache Kafka® for Python Developers

Lucia Cerchie

Kafka Streams Microservices thumbnail
45:48
World Congress 2021 • about 5 years ago

Kafka Streams Microservices

Denis Washington, Olli Salonen

What If We've Been Scaling Stream Processing Wrong All Along? thumbnail
31:06
World Congress 2026 Europe • about 2 months ago

What If We've Been Scaling Stream Processing Wrong All Along?

Hartmut Armbruster

Why and when should we consider Stream Processing frameworks in our solutions thumbnail
30:51
World Congress 2024 • about 2 years ago

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Data binning and understanding histograms thumbnail
30:35
Europe 2026 Virtual • 2 months ago

Data binning and understanding histograms

Michal Bojko

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.