WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Dec 15, 2023

Tips, Techniques, and Common Pitfalls Debugging Kafka

Developersteve

Stop manually piecing together logs during 2 AM Kafka crashes. Zero-code OpenTelemetry auto-instrumentation seamlessly traces asynchronous payloads, instantly isolating execution bottlenecks without refactoring application logic.

Pause
Mute Enter Fullscreen
#1 about 6 min

Introduction to Apache Kafka as architecture glue

An overview of Kafka's origin and its role as event-driven glue in distributed systems.

#2 about 2 min

Real-world Apache Kafka production use cases

How companies like Netflix, Uber, and British Gas utilize Kafka for real-time data streaming.

#3 about 2 min

Comparing Kafka and MQTT for smart devices

The architectural differences between Kafka and MQTT regarding latency, throughput, and packet size.

#4 about 3 min

Building Kafka producers and consumers in Python

How to use Python libraries to build foundational producer and consumer services.

#5 about 3 min

Deep dive into Netflix routing and data streams

Breaking down the complex routing and data streaming layers triggered by platform events.

#6 about 2 min

Advantages of adding Kafka to streaming architecture

The overarching benefits of high capacity, durability, and interoperability that Kafka brings to platforms.

#7 about 4 min

Common challenges in distributed service deployments

Highlighting the complexities of monitoring, load balancing, and scaling clustered applications.

#8 about 3 min

Using distributed tracing instead of server logs

How tracing tools provide contextual error mapping rather than digging through isolated application logs.

#9 about 5 min

Gathering metrics natively with OpenTelemetry frameworks

How the CNCF OpenTelemetry project standardizes application logging models across programming languages.

#10 about 3 min

Simplifying Kubernetes monitoring with Lumigo operators

How to automate instrumentation inside Kubernetes namespaces without modifying deployment code.

#11 about 2 min

Injecting telemetry into Docker and ECS containers

Adding monitoring logic to Python environments within standalone containers using environmental variables.

#12 about 1 min

Automated distributed tracing on AWS Lambda functions

Tracing cloud invocations accurately through simple UI flags and preconfigured Lambda layers.

#13 about 4 min

Setting up the Kafka Python Docker demo

Configuring a multi-broker cluster utilizing Docker Compose, Python, and the Flask web framework.

#14 about 5 min

Deploying the Kafka cluster to Docker Compose

Reviewing Docker deployment flags and executing the containerized background broker nodes.

#15 about 8 min

Executing Flask and exploring live tracing metrics

Simulating application load to generate data traces and examine system runtime performance.

#16 about 2 min

Final takeaways on application telemetry and tracing

The importance of building for scale and standardizing tracing metrics across complex container environments.

#17 about 7 min

Audience questions on brokers and monitoring tools

Discussing Confluent compatibility, maintaining broker metrics, and effective tools to diagnose Kafka stream errors.

Matching moments

3:06 min

Core concepts of Apache Kafka and topic topologies

Denis Washington +1 · World Congress 2021

3:45 min

Reviewing core Apache Kafka architecture and distributed fundamentals

Kirill Kulikov · LIVE

12:33 min

Exploring advanced observability stacks and distributed infrastructure challenges

Pawel Piwosz · LIVE

1:37 min

Introduction to Apache Kafka benchmarking and performance analysis

Kirill Kulikov · LIVE

3:08 min

Live dashboard demonstration of tracing errors and system metrics

Iulia Feroli Iulia Feroli · LIVE

1:13 min

Overcoming integration challenges with Kafka Streams microservices

Denis Washington +1 · World Congress 2021

More from WeAreDevelopers LIVE

WeAreDevelopers LIVE - How DevRel Makes Tech More Human thumbnail
48:06
6 days ago

WeAreDevelopers LIVE - How DevRel Makes Tech More Human

Chris Heilmann, Daniel Cranney

Fake or News: Smart Glasses in Court, Medical Advice Watches & Mushroom Electonics - Jason Torres thumbnail
7:41
7 days ago

Fake or News: Smart Glasses in Court, Medical Advice Watches & Mushroom Electonics - Jason Torres

Chris Heilmann, Daniel Cranney

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
20 days ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Node and Package Security thumbnail
1:08:30
20 days ago

WeAreDevelopers LIVE - Node and Package Security

Chris Heilmann, Daniel Cranney, Zbyszek Tenerowicz

WeAreDevelopers LIVE - Streaming HTML thumbnail
1:10:26
27 days ago

WeAreDevelopers LIVE - Streaming HTML

Chris Heilmann, Daniel Cranney, Jad Joubran

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe thumbnail
1:02:34
about 1 month ago

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe

Chris Heilmann, Daniel Cranney, Justin Halsall, Werner Jeschko, Nicole Jeschko, Mirco Tomić

Upcoming sessions on this topic

Open session

World Congress 2026 North America

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

When Logging Becomes The Outage: Escaping the ECS Logging Trap

Rahul Tanniru

Senior Vice President Of Software Engineering, Jp Morgan Chase

Rahul Tanniru
Open session

World Congress 2026 North America

Taming Rogue Agents: Observability-Driven Evaluation for Production Reliability

Anagha Rumade, Anjana Umapathy, Apoorva Jaiswal

Anagha Rumade
Anjana Umapathy
Apoorva Jaiswal
Open session

World Congress 2026 North America

It’s Alive! Taming the MLOps Franken-Stack: Write, Run, and Serve with Michelangelo

Eric Wang, Paul Zimmerman

Eric Wang
Paul Zimmerman
Open session

World Congress 2026 North America

Replay-Safe Architecture: Building Event-Driven Systems That Can Recover With Confidence

Ishan Shah

PayPal, Software Engineer | Distributed Systems, AI, and Platform Engineering

Ishan Shah
Open session

World Congress 2026 North America

Test Before Release, Enforce at Runtime: Governance for Tool-Using AI Agents

Sachin Gupta

Member of Technical Staff 2 at eBay

Sachin Gupta

Related videos

Let's Get Started With Apache Kafka® for Python Developers thumbnail
38:50
LIVE • over 3 years ago

Let's Get Started With Apache Kafka® for Python Developers

Lucia Cerchie

How to Benchmark Your Apache Kafka thumbnail
35:27
LIVE • over 5 years ago

How to Benchmark Your Apache Kafka

Kirill Kulikov

Kafka Streams Microservices thumbnail
45:48
World Congress 2021 • about 5 years ago

Kafka Streams Microservices

Denis Washington, Olli Salonen

Observability with OpenTelemetry & Elastic thumbnail
30:29
LIVE • almost 2 years ago

Observability with OpenTelemetry & Elastic

Iulia Feroli

Telemetry without the 'Tool Tax' thumbnail
24:30
World Congress 2022 • about 4 years ago

Telemetry without the 'Tool Tax'

Ben Greenberg

The OpenTelemetry mistakes I keep seeing (and how to stop making them) thumbnail
27:02
World Congress 2026 Europe • about 2 months ago

The OpenTelemetry mistakes I keep seeing (and how to stop making them)

Juraci Paixão Kröhling

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.