> Markdown version of [/videos/76-how-to-benchmark-your-apache-kafka?t=116](https://www.wearedevelopers.com/videos/76-how-to-benchmark-your-apache-kafka?t=116). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# How to Benchmark Your Apache Kafka

Why does setting acks=1 sacrifice durability without actually improving end-to-end latency? Uncover the true configuration trade-offs needed to benchmark and optimize your Apache Kafka clusters.

- **Speakers:** Kirill Kulikov
- **Event:** WeAreDevelopers LIVE
- **Published:** November 30, 2020
- **Duration:** 35:27
- **URL:** https://www.wearedevelopers.com/videos/76-how-to-benchmark-your-apache-kafka

## Summary

Benchmarking and tuning Apache Kafka requires carefully balancing inherently competing goals: maximizing throughput and minimizing latency. Because Kafka is a highly complex, distributed event streaming platform, proper configuration demands a deep understanding of its architecture—specifically how producers, consumer groups, and topic partitions handle workloads. Before attempting to tweak parameters or optimize capacity, engineering teams must evaluate their specific production use cases to determine whether their primary service goal revolves around moving massive volumes of data concurrently or delivering individual events as rapidly as possible.

Optimizing for throughput or latency involves distinct configuration trade-offs. To lower end-to-end latency, teams must minimize delays across the produce, publish, commit, and fetch stages. This involves cautiously tuning `linger.ms` and utilizing the sticky partitioner to group non-keyed records into efficient batches. However, common assumptions can be misleading; for instance, setting `acks=1` lowers producer latency but sacrifices data durability without actually improving true end-to-end latency, as consumers must still wait for follower replication before reading messages. Conversely, throughput optimization relies on horizontally scaling partitions and increasing `batch.size` and `fetch.min.bytes` to move larger data chunks simultaneously. Leveraging efficient compression algorithms like LZ4 or ZSTD (while avoiding CPU-heavy GZIP) can also greatly compress network payloads to sustain high data velocity without overwhelming broker resources.

Validating these configurations requires disciplined load testing—including stress, spike, and soak testing—ideally using realistic data payloads. To prevent skewed results, performance tests should never be executed on the same machines hosting the Kafka brokers, and testers must allocate time for JVM warm-up and JIT compilation to avoid artificially high initial latency spikes. Furthermore, relying on mean latency metrics is highly deceptive; tracking the 95th and 99th percentiles offers a much more accurate representation of actual system performance. Engineers can orchestrate these benchmarks and monitor JMX observability metrics using native CLI utilities like `kafka-producer-perf-test`, Kafka’s built-in Trogdor framework for fault injection, or external tools like Apache JMeter paired with the Pepper-Box plugin.

**Keywords:** apache kafka benchmarking, event streaming throughput tuning, kafka latency optimization, producer acknowledgment settings, kafka partition strategies, sticky partitioner, consumer group scaling, kafka load testing techniques, spike and soak testing, jmx observability metrics, jvm warmup considerations, kafka-producer-perf-test, trogdor fault injection, apache jmeter pepper-box, end-to-end latency measurement, percentile metric tracking

## Chapters

1. **Introduction to Apache Kafka benchmarking and performance analysis** (00:19) — Why benchmarking is critical for understanding cluster capacity and optimizing resource utilization.
1. **Reviewing core Apache Kafka architecture and distributed fundamentals** (01:56) — An overview of events, topics, partitions, consumer groups, and the append-only log.
1. **Understanding throughput and latency trade-offs for varied workloads** (05:42) — How workload requirements dictate the service goals when configuring an Apache Kafka cluster.
1. **Measuring end-to-end latency components across the message lifecycle** (07:42) — Breaking down processing time across producing, publishing, replication, and fetching stages.
1. **Tuning producer and consumer configurations for low latency** (09:05) — Adjusting batch size, linger times, acks, and fetch sizes to ensure rapid message delivery.
1. **Reducing end-to-end latency using the sticky partitioner strategy** (13:00) — How the sticky partition strategy improves batching efficiency for non-keyed records to decrease wait times.
1. **Scaling topic partitions and configurations to maximize throughput** (15:37) — Maximizing data transmission rates through partition scaling, batch optimization, and selection of compression algorithms.
1. **Defining objectives and methodologies for systematic load testing** (21:54) — Exploring stress, spike, and soak testing methodologies to evaluate system behavioral limits under heavy use.
1. **Best practices for environment setup and executing load tests** (23:40) — Ensuring accurate results through dedicated test environments, JVM warm-up phases, and interpreting percentile metrics.
1. **Tracking observability metrics to ensure healthy cluster performance** (26:58) — Monitoring hardware usage, system state, and structural JMX metrics to maintain overall cluster health during operation.
1. **Using native command-line tools for synthetic performance testing** (29:37) — Utilizing built-in producer and consumer CLI scripts to generate specific computational loads and measure throughput accurately.
1. **Executing workloads and injecting system faults with Trogdor** (32:13) — Leveraging Apache Kafka's built-in framework to run complex benchmarks and simulate resilient system failures.
1. **Conducting advanced load testing workflows with Apache JMeter** (33:40) — Using the Pepper-Box plugin to finely control request threads, message payload sizes, and construct detailed web dashboards.

## Related Moments

- [Audience questions on brokers and monitoring tools](https://www.wearedevelopers.com/videos/838-tips-techniques-and-common-pitfalls-debugging-kafka) (from "Tips, Techniques, and Common Pitfalls Debugging Kafka")
- [Benchmarking StoteFlow against Kafka Streams](https://www.wearedevelopers.com/videos/100139-what-if-we-ve-been-scaling-stream-processing-wrong-all-along) (from "What If We've Been Scaling Stream Processing Wrong All Along?")
- [Core concepts of Apache Kafka and topic topologies](https://www.wearedevelopers.com/videos/168-kafka-streams-microservices) (from "Kafka Streams Microservices")
- [Defining Apache Kafka and its primary real-time use cases](https://www.wearedevelopers.com/videos/565-let-s-get-started-with-apache-kafka-for-python-developers) (from "Let's Get Started With Apache Kafka® for Python Developers")
- [Real-world Apache Kafka production use cases](https://www.wearedevelopers.com/videos/838-tips-techniques-and-common-pitfalls-debugging-kafka) (from "Tips, Techniques, and Common Pitfalls Debugging Kafka")
- [Introduction to Apache Kafka as architecture glue](https://www.wearedevelopers.com/videos/838-tips-techniques-and-common-pitfalls-debugging-kafka) (from "Tips, Techniques, and Common Pitfalls Debugging Kafka")

## Related Articles

- [Benefits of Using JMeter For Performance Testing](https://www.wearedevelopers.com/magazine/96-benefits-of-using-jmeter-for-performance-testing)
- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)](https://www.wearedevelopers.com/magazine/745-why-event-driven-architecture-isn-t-about-speed-and-when-you-actually-need-it)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)

## Related Jobs

- [Devops Engineer](https://www.wearedevelopers.com/jobs/ext/1940926-devops-engineer) at **Bitpanda**
- [Software Engineer (L2) Segment Team](https://www.wearedevelopers.com/jobs/ext/209570-software-engineer-l2-segment-team) at **Twilio**
- [Software Engineer (L2) Segment Team](https://www.wearedevelopers.com/jobs/ext/238794-software-engineer-l2-segment-team) at **Twilio**
- [Platform Engineer - Mercury Runtime Platform](https://www.wearedevelopers.com/jobs/ext/293235-platform-engineer-mercury-runtime-platform) at **Raiffeisen Bank International AG**
- [Twilio's next Staff Software Engineer](https://www.wearedevelopers.com/jobs/ext/1885964-twilio-s-next-staff-software-engineer) at **Twilio**
- [Artificial Intelligence (AI)](https://www.wearedevelopers.com/jobs/ext/1952055-artificial-intelligence-ai) at **Twilio**