WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe • Jul 9, 2026 • Session details

What If We've Been Scaling Stream Processing Wrong All Along?

Hartmut Armbruster

Stop paying the distribution tax for horizontal scale you never need. Vertical scaling on a single JVM easily handles billions of events daily while slashing latency by 30x.

Pause
Mute Enter Fullscreen
#1 about 3 min

The problem of over-engineering stream processing

Comparing small-scale stream processing to running a simple landing page on a complex Kubernetes cluster.

#2 about 2 min

Overview of current stream processing frameworks

An introduction to the programming models and features provided by Apache Flink and Kafka Streams.

#3 about 3 min

Mechanisms of horizontal scalability

How frameworks split work into task slots, shard state stores, redistribute data, and manage fault tolerance.

#4 about 5 min

The distribution tax of horizontal scaling

The operational complexity, network shuffling, and latency costs associated with distributed systems.

#5 about 6 min

A single-instance vertical scaling architecture

Proposing a new design using virtual threads, global local state, and barrier synchronization to avoid horizontal scaling.

#6 about 5 min

Benefits of local state and virtual threads

How removing data shuffling and adopting virtual threads lowers latency and simplifies concurrent stream processing.

#7 about 2 min

Hardware and availability limitations

The physical limits of scaling up and the requirements for hot standbys to address disaster recovery on single machines.

#8 about 3 min

Benchmarking StoteFlow against Kafka Streams

Reviewing latency, CPU usage, and memory improvements observed when running workloads on the StoteFlow architecture.

#9 about 3 min

Real-world scale limits of single machines

Analyzing financial transaction speeds to demonstrate that peak workloads easily fit within single-node capabilities.

#10 about 3 min

Scaling the Kafka infrastructure versus applications

Why scaling Kafka brokers horizontally remains necessary even if individual stream processors scale vertically.

Matching moments

1:09 min

Evaluating mature stream processing frameworks for production systems

Soroosh Khodami Soroosh Khodami · World Congress 2024

4:30 min

Introducing data management and the shift to streaming

Mary Grygleski Mary Grygleski · LIVE

3:55 min

Infrastructure challenges when combining Kafka with Apache Flink

Bobur Umurzokov · LIVE

1:33 min

Overcoming typical barriers to real-time stream processing adoption

Bobur Umurzokov · LIVE

4:38 min

Operational complexities and performance optimization of stream applications

Denis Washington +1 · World Congress 2021

2:00 min

Moving from traditional databases to decoupled event streaming

Gerard Klijs · LIVE

More from World Congress 2026 Europe

Building the Nervous System of AI - Michael Kagan (NVIDIA) thumbnail
29:17
about 2 months ago

Building the Nervous System of AI - Michael Kagan (NVIDIA)

Michael Kagan, Sead Ahmetović

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote thumbnail
19:41
about 2 months ago

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote

Christian Heilmann

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend thumbnail
32:06
about 2 months ago

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

David vonThenen

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes thumbnail
35:13
about 2 months ago

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes

Irina Branovic

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬 thumbnail
24:23
about 2 months ago

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬

Jeremy Murray

Angstfreude - AI and Corporate Culture - The Thrill and the Threat thumbnail
30:26
about 2 months ago

Angstfreude - AI and Corporate Culture - The Thrill and the Threat

Axel Ebert, Karin Krobath

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 3

Real-Time Data Platforms at Trillion-Event Scale

Diptamay Sanyal

Principal Engineer | Data, AI & Cybersecurity Platforms

Diptamay Sanyal
Open session

World Congress 2026 North America

September 24, 2026 · 17:30–18:00

Stage 3

Scaling Distributed Queues for AI workloads

Jasmit Kaur Saluja

Software Engineer at Meta Platforms Inc

Jasmit Kaur Saluja
Open session

World Congress 2026 North America

September 25, 2026 · 09:00–09:30

Stage 7

AI Agents are Only as Smart as their Context: Building a Real-Time Context Engine at Intuit

Bharat Patel

Lead Software Engineer at Intuit

Bharat Patel
Open session

World Congress 2026 North America

September 24, 2026 · 16:50–17:20

Stage 9

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

September 25, 2026 · 15:30–15:40

Outdoor Stage

Offloading IoT State: Solving the Memory Crisis on Legacy Hardware

Pravin Nagare

Senior Software Engineer, Roku

Pravin Nagare
Open session

World Congress 2026 North America

September 25, 2026 · 09:00–09:30

Stage 3

Your Thread Pool Is Lying to You — Sizing Concurrency from Rate Limits and Latency, Not Guesswork

Ratul Ghosh, Sesha Chennupati

Ratul Ghosh
Sesha Chennupati

Related videos

Why and when should we consider Stream Processing frameworks in our solutions thumbnail
30:51
World Congress 2024 • about 2 years ago

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Kafka Streams Microservices thumbnail
45:48
World Congress 2021 • about 5 years ago

Kafka Streams Microservices

Denis Washington, Olli Salonen

Beyond Loom: Practical Patterns for Virtual Threads, Structured Concurrency and Reactive Interop thumbnail
28:47
Europe 2026 Virtual • 2 months ago

Beyond Loom: Practical Patterns for Virtual Threads, Structured Concurrency and Reactive Interop

Ivan Šarić

From event streaming to event sourcing 101 thumbnail
27:33
LIVE • almost 6 years ago

From event streaming to event sourcing 101

Gerard Klijs

The Rise of Reactive Microservices thumbnail
46:24
World Congress 2022 • about 4 years ago

The Rise of Reactive Microservices

David Leitner

Tips, Techniques, and Common Pitfalls Debugging Kafka thumbnail
54:29
LIVE • over 2 years ago

Tips, Techniques, and Common Pitfalls Debugging Kafka

Developersteve

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.