An overview of how change data capture (CDC) with Debezium and Quarkus can solve the problem of dual writes in microservices.
Dual writes to multiple databases or services can lead to data inconsistencies when one of the writes fails.
Apache Kafka is a fault-tolerant, scalable, publish-subscribe system designed for real-time event stream processing.
Change data capture automatically captures database changes like inserts, updates, and deletes and streams them as events.
Debezium is a Kafka connector that taps into database transaction logs to reliably capture and propagate data changes.
Debezium change events are JSON messages containing before and after states of the data, plus metadata about the operation.
The outbox pattern ensures data consistency by writing business data and an event to an outbox table within a single database transaction.
The strangler fig pattern uses CDC to replicate data from a monolith to a new microservice, enabling a gradual and safe migration.
Use Quarkus to implement the outbox pattern and deploy the entire system, including Kafka managed by Strimzi, on Kubernetes.
A practical demonstration shows how inserting data into a database table automatically triggers Debezium to publish a change event to a Kafka topic.
Discussion covers the challenges of building a custom CDC solution, Debezium's fault tolerance, and handling lost transaction logs.
Bitpanda
Vienna, Austria
57:19Mary Grygleski
44:40Alex Soto
38:50Lucia Cerchie
39:10Niklas Heidloff
33:40Niklas Heidloff
39:16Holly Cummins
37:55Adam Bien
35:27Kirill Kulikov
Jobs that call for the skills explored in this talk.
American IT Systems
Eindhoven, Netherlands
Cardinal Integrated Technologies Inc
Edison, United States of America
Cognizant Technology Solutions Corporation
Boston, United States of America
First Soft Solutions
Jersey City, United States of America
The Depository Trust & Clearing Corporation
Coppell, United States of America
REQUEST TECHNOLOGY
Berkeley, United States of America
