WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe • Jul 9, 2026 • Session details

What If We've Been Scaling Stream Processing Wrong All Along?

Hartmut Armbruster

Stop paying the distribution tax for horizontal scale you never need. Vertical scaling on a single JVM easily handles billions of events daily while slashing latency by 30x.

Pause
Mute Enter Fullscreen
#1 about 3 min

The problem of over-engineering stream processing

Comparing small-scale stream processing to running a simple landing page on a complex Kubernetes cluster.

#2 about 2 min

Overview of current stream processing frameworks

An introduction to the programming models and features provided by Apache Flink and Kafka Streams.

#3 about 3 min

Mechanisms of horizontal scalability

How frameworks split work into task slots, shard state stores, redistribute data, and manage fault tolerance.

#4 about 5 min

The distribution tax of horizontal scaling

The operational complexity, network shuffling, and latency costs associated with distributed systems.

#5 about 6 min

A single-instance vertical scaling architecture

Proposing a new design using virtual threads, global local state, and barrier synchronization to avoid horizontal scaling.

#6 about 5 min

Benefits of local state and virtual threads

How removing data shuffling and adopting virtual threads lowers latency and simplifies concurrent stream processing.

#7 about 2 min

Hardware and availability limitations

The physical limits of scaling up and the requirements for hot standbys to address disaster recovery on single machines.

#8 about 3 min

Benchmarking StoteFlow against Kafka Streams

Reviewing latency, CPU usage, and memory improvements observed when running workloads on the StoteFlow architecture.

#9 about 3 min

Real-world scale limits of single machines

Analyzing financial transaction speeds to demonstrate that peak workloads easily fit within single-node capabilities.

#10 about 3 min

Scaling the Kafka infrastructure versus applications

Why scaling Kafka brokers horizontally remains necessary even if individual stream processors scale vertically.

Matching moments

1:09 min

Evaluating mature stream processing frameworks for production systems

Soroosh Khodami Soroosh Khodami · WWC 2024

4:30 min

Introducing data management and the shift to streaming

Mary Grygleski Mary Grygleski · LIVE

3:55 min

Infrastructure challenges when combining Kafka with Apache Flink

Bobur Umurzokov · LIVE

1:33 min

Overcoming typical barriers to real-time stream processing adoption

Bobur Umurzokov · LIVE

4:38 min

Operational complexities and performance optimization of stream applications

Denis Washington +1 · WWC 2021

2:00 min

Moving from traditional databases to decoupled event streaming

Gerard Klijs · LIVE

More from World Congress 2026 Europe

Building the Nervous System of AI - Michael Kagan (NVIDIA) thumbnail
29:17
about 1 month ago

Building the Nervous System of AI - Michael Kagan (NVIDIA)

Michael Kagan, Sead Ahmetović

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote thumbnail
19:41
about 1 month ago

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote

Christian Heilmann

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend thumbnail
32:06
about 1 month ago

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

David vonThenen

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes thumbnail
35:13
about 1 month ago

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes

Irina Branovic

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬 thumbnail
24:23
about 1 month ago

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬

Jeremy Murray

Angstfreude - AI and Corporate Culture - The Thrill and the Threat thumbnail
30:26
about 1 month ago

Angstfreude - AI and Corporate Culture - The Thrill and the Threat

Axel Ebert, Karin Krobath

Upcoming sessions on this topic

Open session

World Congress 2026 North America

AI Agents are Only as Smart as their Context: Building a Real-Time Context Engine at Intuit

Bharat Patel

Lead Software Engineer at Intuit

Bharat Patel
Open session

World Congress 2026 North America

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

Offloading IoT State: Solving the Memory Crisis on Legacy Hardware

Pravin Nagare

Senior Software Engineer, Roku

Pravin Nagare
Open session

World Congress 2026 North America

Merging at Scale: From Broken Builds to Green Mainline

Manjari Akella, Preetam Dwivedi

Manjari Akella
Preetam Dwivedi
Open session

World Congress 2026 North America

The $0.15/month Algo-Trader: Architecting High-Performance Serverless Trading Engines

Vishwak Thatikonda

Lead Full Stack Engineer at SDH Systems

Vishwak Thatikonda
Open session

World Congress 2026 North America

public void saveMoney(AI): The Developer's Guide to Unit Economics

Hrushikesh Pokala

Senior Software Engineer Lead at Equifax

Hrushikesh Pokala

Related videos

Why and when should we consider Stream Processing frameworks in our solutions thumbnail
30:51
WWC 2024 • almost 2 years ago

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Kafka Streams Microservices thumbnail
45:48
WWC 2021 • about 5 years ago

Kafka Streams Microservices

Denis Washington, Olli Salonen

Beyond Loom: Practical Patterns for Virtual Threads, Structured Concurrency and Reactive Interop thumbnail
28:47
Europe 2026 Virtual • about 1 month ago

Beyond Loom: Practical Patterns for Virtual Threads, Structured Concurrency and Reactive Interop

Ivan Šarić

From event streaming to event sourcing 101 thumbnail
27:33
LIVE • over 5 years ago

From event streaming to event sourcing 101

Gerard Klijs

The Rise of Reactive Microservices thumbnail
46:24
WWC 2022 • about 4 years ago

The Rise of Reactive Microservices

David Leitner

Tips, Techniques, and Common Pitfalls Debugging Kafka thumbnail
54:29
LIVE • over 2 years ago

Tips, Techniques, and Common Pitfalls Debugging Kafka

Developersteve

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.