WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe • Jul 9, 2026 • Session details

What If We've Been Scaling Stream Processing Wrong All Along?

Hartmut Armbruster

Stop paying the distribution tax for horizontal scale you never need. Vertical scaling on a single JVM easily handles billions of events daily while slashing latency by 30x.

Pause
Mute Enter Fullscreen
#1 about 3 min

The problem of over-engineering stream processing

Comparing small-scale stream processing to running a simple landing page on a complex Kubernetes cluster.

#2 about 2 min

Overview of current stream processing frameworks

An introduction to the programming models and features provided by Apache Flink and Kafka Streams.

#3 about 3 min

Mechanisms of horizontal scalability

How frameworks split work into task slots, shard state stores, redistribute data, and manage fault tolerance.

#4 about 5 min

The distribution tax of horizontal scaling

The operational complexity, network shuffling, and latency costs associated with distributed systems.

#5 about 6 min

A single-instance vertical scaling architecture

Proposing a new design using virtual threads, global local state, and barrier synchronization to avoid horizontal scaling.

#6 about 5 min

Benefits of local state and virtual threads

How removing data shuffling and adopting virtual threads lowers latency and simplifies concurrent stream processing.

#7 about 2 min

Hardware and availability limitations

The physical limits of scaling up and the requirements for hot standbys to address disaster recovery on single machines.

#8 about 3 min

Benchmarking StoteFlow against Kafka Streams

Reviewing latency, CPU usage, and memory improvements observed when running workloads on the StoteFlow architecture.

#9 about 3 min

Real-world scale limits of single machines

Analyzing financial transaction speeds to demonstrate that peak workloads easily fit within single-node capabilities.

#10 about 3 min

Scaling the Kafka infrastructure versus applications

Why scaling Kafka brokers horizontally remains necessary even if individual stream processors scale vertically.

Matching moments

1:09 min

Evaluating mature stream processing frameworks for production systems

Soroosh Khodami Soroosh Khodami · World Congress 2024

4:30 min

Introducing data management and the shift to streaming

Mary Grygleski Mary Grygleski · LIVE

3:55 min

Infrastructure challenges when combining Kafka with Apache Flink

Bobur Umurzokov · LIVE

1:33 min

Overcoming typical barriers to real-time stream processing adoption

Bobur Umurzokov · LIVE

4:38 min

Operational complexities and performance optimization of stream applications

Denis Washington +1 · World Congress 2021

2:00 min

Moving from traditional databases to decoupled event streaming

Gerard Klijs · LIVE

More from World Congress 2026 Europe

Building the Nervous System of AI - Michael Kagan (NVIDIA) thumbnail
29:17
3 months ago

Building the Nervous System of AI - Michael Kagan (NVIDIA)

Michael Kagan, Sead Ahmetović

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote thumbnail
19:41
3 months ago

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote

Christian Heilmann

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend thumbnail
32:06
3 months ago

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

David vonThenen

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes thumbnail
35:13
3 months ago

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes

Irina Branovic

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬 thumbnail
24:23
3 months ago

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬

Jeremy Murray

Angstfreude - AI and Corporate Culture - The Thrill and the Threat thumbnail
30:26
3 months ago

Angstfreude - AI and Corporate Culture - The Thrill and the Threat

Axel Ebert, Karin Krobath

Upcoming sessions on this topic

Open session

How We Cut Our API's p99 Latency from Minutes to Under a Second

  • Deepak Agrawal

    Atlassian

    Principal Software Engineer

Open session

We Deleted Our Observability Stack and Rebuilt It with OTEL: 12 Engineers to 4 at 20K+ clusters

  • Yash Sharma

    DigitalOcean

    Developer Advocate

Open session

Multi-Cloud Isn't a Project. It's a Side Effect.

  • Antonio Mendoza Pérez

    Temporal Technologies

    Staff Developer Success Engineer

  • Gaurav Thadani

    Temporal

    Staff Developer Success Engineer

Open session

React Under Fire: Rendering Performance for Streaming Agent UIs

  • Abhirupa Mitra

    Slack

    Senior Member of Technical Staff

Open session

AI Is the Easy Part: Why Faster Engineers Don't Make Faster Businesses

  • Bharat Sharma

    Booking

    Senior Engineering Manager

Open session

Java is Coming for Python’s Lunch: The Rise of JVM AI Agents

  • Frederik Pietzko

    Jetbrains

    Developer Advocate

Related videos

Real-Time Data Platforms at Trillion-Event Scale thumbnail
26:39
World Congress 2026 North America • 19 days ago

Real-Time Data Platforms at Trillion-Event Scale

Diptamay Sanyal

Why and when should we consider Stream Processing frameworks in our solutions thumbnail
30:51
World Congress 2024 • about 2 years ago

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Kafka Streams Microservices thumbnail
45:48
World Congress 2021 • over 5 years ago

Kafka Streams Microservices

Denis Washington, Olli Salonen

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka thumbnail
34:30
World Congress 2026 North America • 19 days ago

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Beyond Loom: Practical Patterns for Virtual Threads, Structured Concurrency and Reactive Interop thumbnail
28:47
Europe 2026 Virtual • 3 months ago

Beyond Loom: Practical Patterns for Virtual Threads, Structured Concurrency and Reactive Interop

Ivan Šarić

From event streaming to event sourcing 101 thumbnail
27:33
LIVE • almost 6 years ago

From event streaming to event sourcing 101

Gerard Klijs

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.