WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Dec 15, 2023

Tips, Techniques, and Common Pitfalls Debugging Kafka

Developersteve

Stop manually piecing together logs during 2 AM Kafka crashes. Zero-code OpenTelemetry auto-instrumentation seamlessly traces asynchronous payloads, instantly isolating execution bottlenecks without refactoring application logic.

Pause
Mute Enter Fullscreen
#1 about 6 min

Introduction to Apache Kafka as architecture glue

An overview of Kafka's origin and its role as event-driven glue in distributed systems.

#2 about 2 min

Real-world Apache Kafka production use cases

How companies like Netflix, Uber, and British Gas utilize Kafka for real-time data streaming.

#3 about 2 min

Comparing Kafka and MQTT for smart devices

The architectural differences between Kafka and MQTT regarding latency, throughput, and packet size.

#4 about 3 min

Building Kafka producers and consumers in Python

How to use Python libraries to build foundational producer and consumer services.

#5 about 3 min

Deep dive into Netflix routing and data streams

Breaking down the complex routing and data streaming layers triggered by platform events.

#6 about 2 min

Advantages of adding Kafka to streaming architecture

The overarching benefits of high capacity, durability, and interoperability that Kafka brings to platforms.

#7 about 4 min

Common challenges in distributed service deployments

Highlighting the complexities of monitoring, load balancing, and scaling clustered applications.

#8 about 3 min

Using distributed tracing instead of server logs

How tracing tools provide contextual error mapping rather than digging through isolated application logs.

#9 about 5 min

Gathering metrics natively with OpenTelemetry frameworks

How the CNCF OpenTelemetry project standardizes application logging models across programming languages.

#10 about 3 min

Simplifying Kubernetes monitoring with Lumigo operators

How to automate instrumentation inside Kubernetes namespaces without modifying deployment code.

#11 about 2 min

Injecting telemetry into Docker and ECS containers

Adding monitoring logic to Python environments within standalone containers using environmental variables.

#12 about 1 min

Automated distributed tracing on AWS Lambda functions

Tracing cloud invocations accurately through simple UI flags and preconfigured Lambda layers.

#13 about 4 min

Setting up the Kafka Python Docker demo

Configuring a multi-broker cluster utilizing Docker Compose, Python, and the Flask web framework.

#14 about 5 min

Deploying the Kafka cluster to Docker Compose

Reviewing Docker deployment flags and executing the containerized background broker nodes.

#15 about 8 min

Executing Flask and exploring live tracing metrics

Simulating application load to generate data traces and examine system runtime performance.

#16 about 2 min

Final takeaways on application telemetry and tracing

The importance of building for scale and standardizing tracing metrics across complex container environments.

#17 about 7 min

Audience questions on brokers and monitoring tools

Discussing Confluent compatibility, maintaining broker metrics, and effective tools to diagnose Kafka stream errors.

Matching moments

3:06 min

Core concepts of Apache Kafka and topic topologies

Denis Washington +1 · World Congress 2021

3:45 min

Reviewing core Apache Kafka architecture and distributed fundamentals

Kirill Kulikov · LIVE

12:33 min

Exploring advanced observability stacks and distributed infrastructure challenges

Pawel Piwosz · LIVE

1:37 min

Introduction to Apache Kafka benchmarking and performance analysis

Kirill Kulikov · LIVE

3:08 min

Live dashboard demonstration of tracing errors and system metrics

Iulia Feroli Iulia Feroli · LIVE

1:13 min

Overcoming integration challenges with Kafka Streams microservices

Denis Washington +1 · World Congress 2021

More from WeAreDevelopers LIVE

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude thumbnail
6:16
9 days ago

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude

Chris Heilmann, Daniel Cranney

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail
5:46
14 days ago

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

WeAreDevelopers - Of Agents and Robots thumbnail
1:04:42
14 days ago

WeAreDevelopers - Of Agents and Robots

Chris Heilmann, Daniel Cranney, Marc Plogas

WeAreDevelopers LIVE - How DevRel Makes Tech More Human thumbnail
48:06
21 days ago

WeAreDevelopers LIVE - How DevRel Makes Tech More Human

Chris Heilmann, Daniel Cranney

Fake or News: Smart Glasses in Court, Medical Advice Watches & Mushroom Electonics - Jason Torres thumbnail
7:41
22 days ago

Fake or News: Smart Glasses in Court, Medical Advice Watches & Mushroom Electonics - Jason Torres

Chris Heilmann, Daniel Cranney

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
about 1 month ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 24, 2026 · 16:50–17:20

Stage 9

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 3

Real-Time Data Platforms at Trillion-Event Scale

Diptamay Sanyal

Principal Engineer | Data, AI & Cybersecurity Platforms

Diptamay Sanyal
Open session

World Congress 2026 North America

September 24, 2026 · 13:30–15:30

Stage 14

From Signal to Action: Empowering Your AI SRE with OpenTelemetry Data

Julia Furst Morgado, Raphael Manke

Julia Furst Morgado
Raphael Manke
Open session

World Congress 2026 North America

September 25, 2026 · 11:00–11:30

Outdoor Stage

Loop Engineering: Designing and Observing the Loops That Run Your Agents With Opik

Abigail Morgan

AI Developer Advocate at Comet

Abigail Morgan
Open session

World Congress 2026 North America

September 24, 2026 · 14:50–15:20

Stage 4

When Logging Becomes The Outage: Escaping the ECS Logging Trap

Rahul Tanniru

Senior Vice President Of Software Engineering, Jp Morgan Chase

Rahul Tanniru
Open session

World Congress 2026 North America

September 24, 2026 · 11:40–12:10

Stage 4

Taming Rogue Agents: Observability-Driven Evaluation for Production Reliability

Anagha Rumade, Anjana Umapathy, Apoorva Jaiswal

Anagha Rumade
Anjana Umapathy
Apoorva Jaiswal

Related videos

Let's Get Started With Apache Kafka® for Python Developers thumbnail
38:50
LIVE • over 3 years ago

Let's Get Started With Apache Kafka® for Python Developers

Lucia Cerchie

How to Benchmark Your Apache Kafka thumbnail
35:27
LIVE • almost 6 years ago

How to Benchmark Your Apache Kafka

Kirill Kulikov

Kafka Streams Microservices thumbnail
45:48
World Congress 2021 • about 5 years ago

Kafka Streams Microservices

Denis Washington, Olli Salonen

Observability with OpenTelemetry & Elastic thumbnail
30:29
LIVE • almost 2 years ago

Observability with OpenTelemetry & Elastic

Iulia Feroli

Telemetry without the 'Tool Tax' thumbnail
24:30
World Congress 2022 • about 4 years ago

Telemetry without the 'Tool Tax'

Ben Greenberg

The OpenTelemetry mistakes I keep seeing (and how to stop making them) thumbnail
27:02
World Congress 2026 Europe • 2 months ago

The OpenTelemetry mistakes I keep seeing (and how to stop making them)

Juraci Paixão Kröhling

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.