> Markdown version of [/videos/168-kafka-streams-microservices?t=259](https://www.wearedevelopers.com/videos/168-kafka-streams-microservices?t=259). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Kafka Streams Microservices

Replacing tightly coupled microservices with Kafka Streams guarantees continuous data availability. Learn to build resilient, event-sourced architectures by treating Kafka as primary persistent storage.

- **Speakers:** Denis Washington, Olli Salonen
- **Event:** World Congress 2021
- **Published:** June 30, 2021
- **Duration:** 45:48
- **URL:** https://www.wearedevelopers.com/videos/168-kafka-streams-microservices

## Summary

Replacing tightly coupled microservices with an event-sourced architecture using Apache Kafka and Kafka Streams fundamentally alters system robustness and availability. Faced with integrating volatile on-premise legacy systems into a resilient cloud platform, engineering teams can eliminate cross-service synchronous queries by treating Kafka topics as primary, persistent storage. By streaming all business entities—such as dynamic product catalogs and user inventories—into distinct event pipelines, services build local materialized views. This decoupling guarantees continuous data availability for clients, even if backend components suffer momentary faults or extended downtime.

Implementing distributed state manipulation across pipelines highlights both the distinct power and complexity of stream processing. Real-world applications, such as managing unique user constraints in inventory systems or opportunistically registering warranty extensions from aggregate streams, require explicit management of eventual consistency. Developers often encounter sophisticated race conditions and partitioning mismatches when standard key-value routing diverges from local state partitions. Mitigating these issues involves implementing repartitioning topologies and manually controlling synchronous state stores backed by internal changelog topics, demonstrating that stream design requires rigorous planning around keys, parallel partitions, and state locality.

While a pure Kafka-driven topology affords immense temporal flexibility—such as retroactively applying new business logic to historical event logs—it introduces steep operational overhead. Critical lessons emphasize the danger of default configurations, where balancing throughput versus latency or untangling uncommitted read errors in transactional flows can break downstream integrity. Transitioning heavily monolithic streams into granular, sub-topology microservices prevents configuration pollution and accelerates deployments. Ultimately, mature architectures benefit significantly from a hybrid approach, augmenting stream purity with locally embedded relational databases and change pattern connectors to simplify strict transactional logic while preserving an ecosystem-wide event log.

**Keywords:** apache kafka, kafka streams, event sourcing patterns, microservices architecture, materialized views, eventual consistency challenges, distributed state stores, stream topology partitioning, repartitioning steps, transactional message producers, changelog topics, fault-tolerant data pipelines, change data capture, legacy system integration, synchronous api decoupling, hybrid stream architecture

## Chapters

1. **Overcoming integration challenges with Kafka Streams microservices** (00:00) — Designing a robust platform to integrate disconnected cloud systems without building monolithic data silos.
1. **Core concepts of Apache Kafka and topic topologies** (01:13) — Understanding the fundamental mechanics of distributed event logs, consumer offsets, and stream topology abstractions.
1. **Transitioning from synchronous microservices to event sourcing** (04:19) — Replacing direct commands and system querying with primary event streams to eliminate tight coupling.
1. **Designing stream topologies and managing system failure scenarios** (09:57) — Leveraging stored event logs to replay historical data and maintain data read availability during outages.
1. **Building a consistent product catalog stream data pipeline** (13:14) — Aggregating, cleaning, and enriching unreliable inbound event streams into an interactive full-text search view.
1. **Solving race conditions and distributed state with repartitioning** (17:06) — Restructuring Kafka topics and IDs to enforce synchronous property validations across concurrent backend processors.
1. **Automating system behavior across independent event data streams** (25:33) — Writing processor topologies that combine separate state entities to trigger automated actions without API orchestration.
1. **Operational complexities and performance optimization of stream applications** (28:46) — Overcoming deployment pitfalls arising from transactional properties and overriding shared configurations under heavy loads.
1. **Blending traditional relational databases with event driven streams** (33:25) — Hooking classic database engines up to log publisher connectors to simplify complex unique entity logic.
1. **Exploring query boundaries, data storage, and architecture limits** (34:49) — Clarifying the practical boundaries of local processing safety, testing frameworks, and scalable long-term message compaction.

## Related Moments

- [Moving from traditional databases to decoupled event streaming](https://www.wearedevelopers.com/videos/91-from-event-streaming-to-event-sourcing-101) (from "From event streaming to event sourcing 101")
- [Reviewing core Apache Kafka architecture and distributed fundamentals](https://www.wearedevelopers.com/videos/76-how-to-benchmark-your-apache-kafka) (from "How to Benchmark Your Apache Kafka")
- [Introducing data management and the shift to streaming](https://www.wearedevelopers.com/videos/538-event-messaging-and-streaming-with-apache-pulsar) (from "Event Messaging and Streaming with Apache Pulsar")
- [Overview of current stream processing frameworks](https://www.wearedevelopers.com/videos/100139-what-if-we-ve-been-scaling-stream-processing-wrong-all-along) (from "What If We've Been Scaling Stream Processing Wrong All Along?")
- [Advantages of adding Kafka to streaming architecture](https://www.wearedevelopers.com/videos/838-tips-techniques-and-common-pitfalls-debugging-kafka) (from "Tips, Techniques, and Common Pitfalls Debugging Kafka")
- [Introduction to Apache Kafka as architecture glue](https://www.wearedevelopers.com/videos/838-tips-techniques-and-common-pitfalls-debugging-kafka) (from "Tips, Techniques, and Common Pitfalls Debugging Kafka")

## Related Articles

- [Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)](https://www.wearedevelopers.com/magazine/745-why-event-driven-architecture-isn-t-about-speed-and-when-you-actually-need-it)
- [Why You Shouldn’t Build a Microservice Architecture](https://www.wearedevelopers.com/magazine/118-why-you-shouldn-t-build-a-microservice-architecture)
- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)

## Related Jobs

- [Platform Engineer - Mercury Runtime Platform](https://www.wearedevelopers.com/jobs/ext/293235-platform-engineer-mercury-runtime-platform) at **Raiffeisen Bank International AG**
- [Software Engineer](https://www.wearedevelopers.com/jobs/ext/1304791-software-engineer) at **Bitpanda**
- [Senior DevOps Consultant](https://www.wearedevelopers.com/jobs/ext/1247667-senior-devops-consultant) at **Aok Systems Gmbh**
- [Lead Software Engineer - Data Engineering](https://www.wearedevelopers.com/jobs/ext/2000968-lead-software-engineer-data-engineering) at **Dynatrace**
- [Principal Software Engineer](https://www.wearedevelopers.com/jobs/ext/149130-principal-software-engineer) at **Twilio**
- [Senior Software Engineer (m/w/d)](https://www.wearedevelopers.com/jobs/47753-senior-software-engineer-m-w-d) at **WALTER GROUP**