> Markdown version of [/videos/100139-what-if-we-ve-been-scaling-stream-processing-wrong-all-along](https://www.wearedevelopers.com/videos/100139-what-if-we-ve-been-scaling-stream-processing-wrong-all-along). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# What If We've Been Scaling Stream Processing Wrong All Along?

Stop paying the distribution tax for horizontal scale you never need. Vertical scaling on a single JVM easily handles billions of events daily while slashing latency by 30x.

- **Speakers:** [Hartmut Armbruster](https://www.wearedevelopers.com/@hartmut-armbruster)
- **Event:** World Congress 2026 Europe
- **Published:** July 9, 2026
- **Duration:** 31:06
- **URL:** https://www.wearedevelopers.com/videos/100139-what-if-we-ve-been-scaling-stream-processing-wrong-all-along

## Summary

The uncomfortable truth about modern stream processing frameworks like Apache Flink and Kafka Streams is that developers often pay a severe 'distribution tax' for horizontal scalability they will never actually need. Driven by assumptions of Uber-level data volumes, engineering teams accept network shuffles, redundant serialization, state migrations, and complex rebalancing storms as operational norms. Yet, even massive real-world global systems—like SWIFT financial messaging or Visa payment processing—operate at roughly 10,000 operations per second, a volume completely out of sync with the distributed complexities forced upon them.

By questioning this default and embracing vertical scaling, software architects can fundamentally simplify their data pipelines. Shifting to a single-instance architecture powered by Java Project Loom allows applications to manage extreme concurrency through virtual threads without blocking main CPU execution. This approach leverages a central dispatcher utilizing Chandy-Lamport-style commit barriers, global local state stores backed by RocksDB, and exactly-once Kafka transactional guarantees. Because the data remains on a single JVM and state access rules are centralized, cross-network shuffling and complex partition-based broadcasting are completely eradicated from the workflow.

The performance outcomes of halting default distribution validate the efficiency of vertical scaling. Benchmarks of this single-instance approach achieved processing capacities approaching 10 billion events daily (300MB/s) on a standard 8-core virtual machine. Eliminating the distribution tax allowed the system to deliver up to a 30x reduction in P99 latency, drastically lower heap demands, and vastly simpler deterministic deployments. Ultimately, aligning engineering architecture with realistic data thresholds yields faster, cheaper, and more robust streaming solutions without requiring highly specialized teams to babysit them in production.

**Keywords:** stream processing architecture, horizontal vs vertical scaling, apache flink workflows, kafka streams optimization, distributed systems tax, java project loom, single-instance deployment, virtual threads concurrency, chandy-lamport barriers, exactly-once semantics, consumer rebalancing storms, stateful stream aggregations, global state management, rocksdb performance tuning, network shuffle bottlenecks

## Chapters

1. **The problem of over-engineering stream processing** (01:05) — Comparing small-scale stream processing to running a simple landing page on a complex Kubernetes cluster.
1. **Overview of current stream processing frameworks** (03:12) — An introduction to the programming models and features provided by Apache Flink and Kafka Streams.
1. **Mechanisms of horizontal scalability** (05:09) — How frameworks split work into task slots, shard state stores, redistribute data, and manage fault tolerance.
1. **The distribution tax of horizontal scaling** (07:45) — The operational complexity, network shuffling, and latency costs associated with distributed systems.
1. **A single-instance vertical scaling architecture** (12:34) — Proposing a new design using virtual threads, global local state, and barrier synchronization to avoid horizontal scaling.
1. **Benefits of local state and virtual threads** (17:52) — How removing data shuffling and adopting virtual threads lowers latency and simplifies concurrent stream processing.
1. **Hardware and availability limitations** (22:36) — The physical limits of scaling up and the requirements for hot standbys to address disaster recovery on single machines.
1. **Benchmarking StoteFlow against Kafka Streams** (24:08) — Reviewing latency, CPU usage, and memory improvements observed when running workloads on the StoteFlow architecture.
1. **Real-world scale limits of single machines** (26:32) — Analyzing financial transaction speeds to demonstrate that peak workloads easily fit within single-node capabilities.
1. **Scaling the Kafka infrastructure versus applications** (29:04) — Why scaling Kafka brokers horizontally remains necessary even if individual stream processors scale vertically.

## Related Moments

- [Evaluating mature stream processing frameworks for production systems](https://www.wearedevelopers.com/videos/1085-why-and-when-should-we-consider-stream-processing-frameworks-in-our-solutions) (from "Why and when should we consider Stream Processing frameworks in our solutions")
- [Introducing data management and the shift to streaming](https://www.wearedevelopers.com/videos/538-event-messaging-and-streaming-with-apache-pulsar) (from "Event Messaging and Streaming with Apache Pulsar")
- [Infrastructure challenges when combining Kafka with Apache Flink](https://www.wearedevelopers.com/videos/1233-python-based-data-streaming-pipelines-within-minutes) (from "Python-Based Data Streaming Pipelines Within Minutes")
- [Overcoming typical barriers to real-time stream processing adoption](https://www.wearedevelopers.com/videos/1233-python-based-data-streaming-pipelines-within-minutes) (from "Python-Based Data Streaming Pipelines Within Minutes")
- [Operational complexities and performance optimization of stream applications](https://www.wearedevelopers.com/videos/168-kafka-streams-microservices) (from "Kafka Streams Microservices")
- [Moving from traditional databases to decoupled event streaming](https://www.wearedevelopers.com/videos/91-from-event-streaming-to-event-sourcing-101) (from "From event streaming to event sourcing 101")

## Related Articles

- [Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)](https://www.wearedevelopers.com/magazine/745-why-event-driven-architecture-isn-t-about-speed-and-when-you-actually-need-it)
- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [Now is the time for industrialized software development](https://www.wearedevelopers.com/magazine/601-now-is-the-time-for-industrialized-software-development)
- [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix)

## Related Jobs

- [Principal Software Engineer](https://www.wearedevelopers.com/jobs/ext/149130-principal-software-engineer) at **Twilio**
- [Software Engineer (L2) Segment Team](https://www.wearedevelopers.com/jobs/ext/209570-software-engineer-l2-segment-team) at **Twilio**
- [Software Engineer (L2) Segment Team](https://www.wearedevelopers.com/jobs/ext/238794-software-engineer-l2-segment-team) at **Twilio**
- [Software Engineer (L3) Data Substrate](https://www.wearedevelopers.com/jobs/ext/494111-software-engineer-l3-data-substrate) at **Twilio**
- [Software Engineer (L3) Data Substrate](https://www.wearedevelopers.com/jobs/ext/486993-software-engineer-l3-data-substrate) at **Twilio**
- [Software Engineer (L3) Data Substrate](https://www.wearedevelopers.com/jobs/ext/461301-software-engineer-l3-data-substrate) at **Twilio**