WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • May 12, 2021

Don't Change the Partition Count for Kafka Topics!

Dainius Jocas

A routine increase in Kafka partitions caused Vinted's Elasticsearch to silently serve stale data. Discover why you should never change partition counts on live, offset-dependent topics.

Pause
Mute Enter Fullscreen
#1 about 4 min

Introduction to the data indexing pipeline

Moving primary data from MySQL to Elasticsearch using Kafka creates a scalable indexing pipeline.

#2 about 2 min

Understanding Elasticsearch and optimistic concurrency control

Using optimistic concurrency control with document version numbers ensures only the newest updates are searchable.

#3 about 3 min

Kafka log compaction and tombstone messages

Configuring topics with infinite retention and log compaction prevents disk exhaustion while enabling safe re-indexing.

#4 about 2 min

Connecting systems safely using Kafka Connect

Using Kafka partition offsets as document versions allows Kafka Connect to safely parallelize indexing.

#5 about 3 min

Investigating reports of stale data in Elasticsearch

A production bug report reveals that tombstone messages are failing to delete obsolete Elasticsearch documents.

#6 about 4 min

Comparing Elasticsearch versions and Kafka offsets

Tracing the issue reveals older Elasticsearch documents possessing higher version numbers than newer Kafka offsets.

#7 about 2 min

Identifying the partition count root cause

Consulting documentation and metrics reveals that increasing the Kafka topic partition count triggered the inconsistency.

#8 about 4 min

How partition changes break message ordering

Altering the partition count changes message hashing, destroying the order guarantees for specific keys.

#9 about 3 min

Fixing the data inconsistency with full ingestion

Resolving the data inconsistency requires fully re-ingesting primary datastore records into newly created Kafka topics.

#10 about 2 min

Establishing best practices for Kafka partition sizing

Setting sensible default partition counts and avoiding modifications prevents data loss when relying on message ordering.

Matching moments

2:20 min

Event streaming and distributed systems with Apache Kafka

Sandon Jacobs Sandon Jacobs · World Congress 2026 North America

3:45 min

Reviewing core Apache Kafka architecture and distributed fundamentals

Kirill Kulikov · LIVE

3:06 min

Core concepts of Apache Kafka and topic topologies

Denis Washington +1 · World Congress 2021

2:33 min

Hidden costs of self-hosting and managed Kafka solutions

Bobur Umurzokov · LIVE

8:27 min

Solving race conditions and distributed state with repartitioning

Denis Washington +1 · World Congress 2021

2:00 min

Moving from traditional databases to decoupled event streaming

Gerard Klijs · LIVE

More from WeAreDevelopers LIVE

Read more, Prompt Less thumbnail
1:01:30
about 12 hours ago

Read more, Prompt Less

Chris Heilmann, Daniel Cranney, Luise Freese

WeAreDevelopers LIVE - Tech Education in Two Languages thumbnail
1:00:18
7 days ago

WeAreDevelopers LIVE - Tech Education in Two Languages

Daniel Cranney, Roxy Rodriguez-Becker

WeAreDevelopers LIVE - Playing in Production thumbnail
1:06:40
14 days ago

WeAreDevelopers LIVE - Playing in Production

Andrew MacLean, Daniel Cranney, Christian Heilmann

WeAreDevelopers LIVE - Is CSS Catching Up to Flash? thumbnail
1:03:33
28 days ago

WeAreDevelopers LIVE - Is CSS Catching Up to Flash?

Chris Heilmann, Daniel Cranney, Bramus Van Damme

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along thumbnail
1:21:17
about 1 month ago

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along

Chris Heilmann, Daniel Cranney, Krzysztof Cieślak

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak thumbnail
4:23
about 1 month ago

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

How We Cut Our API's p99 Latency from Minutes to Under a Second

  • Deepak Agrawal

    Atlassian

    Principal Software Engineer

Open session

We Deleted Our Observability Stack and Rebuilt It with OTEL: 12 Engineers to 4 at 20K+ clusters

  • Yash Sharma

    DigitalOcean

    Developer Advocate

Open session

What I Got Wrong Shipping an MCP Server for Live Infrastructure

  • Pritesh Kiri

    Harness

    Developer Relations Engineer

Open session

How 8 GPUs Became 16: Rethinking LLM Inference Through Parallelism, Memory, and Runtime Design

  • Purnima Kumari

    DigitalOCean

    Sr. Solutions Architect II

Open session

Multi-Cloud Isn't a Project. It's a Side Effect.

  • Antonio Mendoza Pérez

    Temporal Technologies

    Staff Developer Success Engineer

  • Gaurav Thadani

    Temporal

    Staff Developer Success Engineer

Open session

Shipping MCP to Real Users: Scaling, Auth, and the PII Leak You Didn't See Coming

  • Hadar Geva

    Myop

    CTO & Co-founder

Related videos

Kafka Streams Microservices thumbnail
45:48
World Congress 2021 • over 5 years ago

Kafka Streams Microservices

Denis Washington, Olli Salonen

What If We've Been Scaling Stream Processing Wrong All Along? thumbnail
31:06
World Congress 2026 Europe • 3 months ago

What If We've Been Scaling Stream Processing Wrong All Along?

Hartmut Armbruster

Let's Get Started With Apache Kafka® for Python Developers thumbnail
38:50
LIVE • over 3 years ago

Let's Get Started With Apache Kafka® for Python Developers

Lucia Cerchie

How to Benchmark Your Apache Kafka thumbnail
35:27
LIVE • almost 6 years ago

How to Benchmark Your Apache Kafka

Kirill Kulikov

Distributed search under the hood thumbnail
46:58
LIVE • about 5 years ago

Distributed search under the hood

Alexander Reelsen

Real-Time Data Platforms at Trillion-Event Scale thumbnail
26:39
World Congress 2026 North America • 20 days ago

Real-Time Data Platforms at Trillion-Event Scale

Diptamay Sanyal

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.