WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe • Jul 9, 2026 • Session details

What If We've Been Scaling Stream Processing Wrong All Along?

Hartmut Armbruster

Stop paying the distribution tax for horizontal scale you never need. Vertical scaling on a single JVM easily handles billions of events daily while slashing latency by 30x.

Pause
Mute Enter Fullscreen
#1 about 3 min

The problem of over-engineering stream processing

Comparing small-scale stream processing to running a simple landing page on a complex Kubernetes cluster.

#2 about 2 min

Overview of current stream processing frameworks

An introduction to the programming models and features provided by Apache Flink and Kafka Streams.

#3 about 3 min

Mechanisms of horizontal scalability

How frameworks split work into task slots, shard state stores, redistribute data, and manage fault tolerance.

#4 about 5 min

The distribution tax of horizontal scaling

The operational complexity, network shuffling, and latency costs associated with distributed systems.

#5 about 6 min

A single-instance vertical scaling architecture

Proposing a new design using virtual threads, global local state, and barrier synchronization to avoid horizontal scaling.

#6 about 5 min

Benefits of local state and virtual threads

How removing data shuffling and adopting virtual threads lowers latency and simplifies concurrent stream processing.

#7 about 2 min

Hardware and availability limitations

The physical limits of scaling up and the requirements for hot standbys to address disaster recovery on single machines.

#8 about 3 min

Benchmarking StoteFlow against Kafka Streams

Reviewing latency, CPU usage, and memory improvements observed when running workloads on the StoteFlow architecture.

#9 about 3 min

Real-world scale limits of single machines

Analyzing financial transaction speeds to demonstrate that peak workloads easily fit within single-node capabilities.

#10 about 3 min

Scaling the Kafka infrastructure versus applications

Why scaling Kafka brokers horizontally remains necessary even if individual stream processors scale vertically.

Matching moments

1:09 min

Evaluating mature stream processing frameworks for production systems

Soroosh Khodami Soroosh Khodami · World Congress 2024

4:30 min

Introducing data management and the shift to streaming

Mary Grygleski Mary Grygleski · LIVE

3:55 min

Infrastructure challenges when combining Kafka with Apache Flink

Bobur Umurzokov · LIVE

1:33 min

Overcoming typical barriers to real-time stream processing adoption

Bobur Umurzokov · LIVE

4:38 min

Operational complexities and performance optimization of stream applications

Denis Washington +1 · World Congress 2021

2:00 min

Moving from traditional databases to decoupled event streaming

Gerard Klijs · LIVE

More from World Congress 2026 Europe

Building the Nervous System of AI - Michael Kagan (NVIDIA) thumbnail
29:17
3 months ago

Building the Nervous System of AI - Michael Kagan (NVIDIA)

Michael Kagan, Sead Ahmetović

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote thumbnail
19:41
3 months ago

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote

Christian Heilmann

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend thumbnail
32:06
3 months ago

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

David vonThenen

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes thumbnail
35:13
3 months ago

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes

Irina Branovic

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬 thumbnail
24:23
3 months ago

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬

Jeremy Murray

Angstfreude - AI and Corporate Culture - The Thrill and the Threat thumbnail
30:26
3 months ago

Angstfreude - AI and Corporate Culture - The Thrill and the Threat

Axel Ebert, Karin Krobath

Related videos

Why and when should we consider Stream Processing frameworks in our solutions thumbnail
30:51
World Congress 2024 • about 2 years ago

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Kafka Streams Microservices thumbnail
45:48
World Congress 2021 • about 5 years ago

Kafka Streams Microservices

Denis Washington, Olli Salonen

Beyond Loom: Practical Patterns for Virtual Threads, Structured Concurrency and Reactive Interop thumbnail
28:47
Europe 2026 Virtual • 3 months ago

Beyond Loom: Practical Patterns for Virtual Threads, Structured Concurrency and Reactive Interop

Ivan Šarić

From event streaming to event sourcing 101 thumbnail
27:33
LIVE • almost 6 years ago

From event streaming to event sourcing 101

Gerard Klijs

The Rise of Reactive Microservices thumbnail
46:24
World Congress 2022 • over 4 years ago

The Rise of Reactive Microservices

David Leitner

Tips, Techniques, and Common Pitfalls Debugging Kafka thumbnail
54:29
LIVE • almost 3 years ago

Tips, Techniques, and Common Pitfalls Debugging Kafka

Developersteve

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.