Dainius Jocas

Don't Change the Partition Count for Kafka Topics!

A well-intentioned infrastructure change silently corrupted our search index. Discover how increasing a Kafka topic's partition count can break your entire data pipeline.

Don't Change the Partition Count for Kafka Topics!
#1about 5 minutes

An overview of the data indexing pipeline architecture

The system moves data from a MySQL primary data store to an Elasticsearch search server using a Kafka and Kafka Connect pipeline.

#2about 1 minute

Using Kafka partition offset for optimistic concurrency control

The system leverages the Kafka partition offset as the document version number in Elasticsearch to enable parallel indexing without data consistency issues.

#3about 2 minutes

Investigating a mysterious data deletion failure in production

A bug report about Elasticsearch failing to delete documents, which serves stale data, could not be reproduced in local or testing environments.

#4about 5 minutes

Discovering the offset and version number mismatch

Manual inspection reveals that the document version in Elasticsearch is significantly higher than the new message offset in the Kafka topic for the same key.

#5about 4 minutes

How changing partition count breaks message ordering guarantees

Increasing the Kafka topic's partition count changes the key hashing algorithm, causing new messages for the same key to land in different partitions with lower offsets.

#6about 4 minutes

The solution and key lessons for managing Kafka topics

The fix required a full data re-ingestion into a new Kafka topic, highlighting the lesson to never increase partition count when message ordering is critical.

Related jobs
Jobs that call for the skills explored in this talk.

Matching moments

Exploring Kafka's core concepts of events, topics, and partitions
01:45 MIN

Exploring Kafka's core concepts of events, topics, and partitions

Let's Get Started With Apache Kafka® for Python Developers

Recapping Kafka's capabilities for real-time data feeds
01:17 MIN

Recapping Kafka's capabilities for real-time data feeds

Let's Get Started With Apache Kafka® for Python Developers

Navigating the Kafka ecosystem and the power of community
03:30 MIN

Navigating the Kafka ecosystem and the power of community

Let's Get Started With Apache Kafka® for Python Developers

Answering questions on Kafka use cases, careers, and learning
22:41 MIN

Answering questions on Kafka use cases, careers, and learning

Let's Get Started With Apache Kafka® for Python Developers

Common challenges of running Kafka at scale
05:28 MIN

Common challenges of running Kafka at scale

Tips, Techniques, and Common Pitfalls Debugging Kafka

A traditional approach to streaming with Kafka and Debezium
04:23 MIN

A traditional approach to streaming with Kafka and Debezium

Python-Based Data Streaming Pipelines Within Minutes

Decoupling microservices with event streams
03:41 MIN

Decoupling microservices with event streams

From event streaming to event sourcing 101

Managing data consistency with change data capture
01:34 MIN

Managing data consistency with change data capture

Software Engineering Social Connection: Yubo’s lean approach to scaling an 80M-user infrastructure

Featured Partners

Related Videos

See all videos
Tips, Techniques, and Common Pitfalls Debugging Kafka54:29

Tips, Techniques, and Common Pitfalls Debugging Kafka

DeveloperSteve

Let's Get Started With Apache Kafka® for Python Developers38:50

Let's Get Started With Apache Kafka® for Python Developers

Lucia Cerchie

How to Benchmark Your Apache Kafka35:27

How to Benchmark Your Apache Kafka

Kirill Kulikov

Kafka Streams Microservices45:48

Kafka Streams Microservices

Denis Washington & Olli Salonen

Why and when should we consider Stream Processing frameworks in our solutions30:51

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Practical Change Data Streaming Use Cases With Debezium And Quarkus52:15

Practical Change Data Streaming Use Cases With Debezium And Quarkus

Alex Soto

Distributed search under the hood46:58

Distributed search under the hood

Alexander Reelsen

From event streaming to event sourcing 10127:33

From event streaming to event sourcing 101

Gerard Klijs

Related Articles

View all articles
CH
Chris Heilmann
Dev Digest 134 - Where pixels sing?
News and ArticlesWeAreDevelopers LIVE Data and Security Day is on Wednesday, 25/09/2024. Learn about OPC UA Updates, Best Practices for Using GitHub Secrets, Passwordless Web 1.5, Emerging AI Security Risks, Data Privacy in LLMs and get a chance to t...
Dev Digest 134 - Where pixels sing?
DC
Daniel Cranney
Dev Digest 168: Hacking Postgres, Blocking Meta and Fixing CSS
Inside last week’s Dev Digest 168 . 📊 The state of OpenAI’s GPT models 🤖 20% of Salesforce code written by AI 👩‍💻 Hacking Postgres 🙅‍♂️ How to block Meta AI from your Instagram 🔧 How to fix common CSS mistakes 💻 Make your GitHub profile stand out 🥱 ...
Dev Digest 168: Hacking Postgres, Blocking Meta and Fixing CSS

From learning to earning

Jobs that call for the skills explored in this talk.

DataOps Kafka

DataOps Kafka

KLETA
Paris, France

Senior
Kafka
DevOps
Continuous Integration