WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2024 • Aug 20, 2024 • Session details

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Are you over-engineering your data pipeline? Learn when to leverage complex stream processing frameworks like Apache Flink, and when simple SQL micro-batching is the smarter architectural choice.

Pause
Mute Enter Fullscreen
#1 about 3 min

Defining stream data processing versus standard event processing

Differentiating continuous data aggregation from discrete message decision making.

#2 about 3 min

Handling message order with event and processing time

Analyzing data by actual event occurrence to guarantee offline state consistency.

#3 about 3 min

Ensuring exactly once message delivery in distributed systems

Utilizing stream framework checkpoint mechanisms to completely avoid duplicate data processing.

#4 about 3 min

Constructing data pipelines with stream processing architecture operators

Connecting basic operator functions to correctly read, transform, and sync continuous inputs.

#5 about 6 min

Applying windowing functions to segment unbounded data streams

Segmenting unbounded continuous data streams using distinct tumbling, sliding, and session windows.

#6 about 4 min

Joining data streams and managing stateful memory processing

Combining separate message sources and retaining critical memory state for complex temporal analytics.

#7 about 2 min

Evaluating mature stream processing frameworks for production systems

Reviewing industry-standard data tools like Apache Flink and Beam for scalable application architecture.

#8 about 5 min

Comparing performance and complexity against standard microservice architectures

Benchmarking Apache Beam data pipelines against Spring Boot to assess throughput and setup overhead.

#9 about 3 min

Assessing the organizational cost of stream framework adoption

Weighing high execution speed against steep learning curves and heavy engineering maintenance bandwidth.

#10 about 1 min

Identifying real-world applications for stream processing technology usage

Exploring practical data implementations across gaming anti-cheat systems, telecommunications, and financial fraud detection.

#11 about 2 min

Establishing adoption prerequisites and business stakeholder strategic alignment

Ensuring transparent business requirements exist before committing complex engineering logic to pipeline structural changes.

Matching moments

1:33 min

Overcoming typical barriers to real-time stream processing adoption

Bobur Umurzokov · LIVE

1:56 min

Overview of current stream processing frameworks

Hartmut Armbruster Hartmut Armbruster · World Congress 2026 Europe

4:14 min

Recognizing architectural drivers pushing event streaming system adoption

Mary Grygleski Mary Grygleski · LIVE

4:30 min

Introducing data management and the shift to streaming

Mary Grygleski Mary Grygleski · LIVE

1:32 min

Comparing stream processing architecture to traditional batch processing

Bobur Umurzokov · LIVE

4:38 min

Operational complexities and performance optimization of stream applications

Denis Washington +1 · World Congress 2021

More from World Congress 2024

Beyond the Hype: Real-World AI Strategies Panel thumbnail
24:07
12 months ago

Beyond the Hype: Real-World AI Strategies Panel

Mike Butcher, Jürgen Müller, Katrin Lehmann, Tobias Regenfuss

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation thumbnail
21:17
almost 2 years ago

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation

Carl Lapierre

Coding an Immersive Copilot using Unity / .NET and Azure OpenAI! thumbnail
31:50
about 2 years ago

Coding an Immersive Copilot using Unity / .NET and Azure OpenAI!

Zaid Zaim

Building a diverse tech company: The power of Diversity, Equity, Inclusion, and Belonging thumbnail
29:43
about 2 years ago

Building a diverse tech company: The power of Diversity, Equity, Inclusion, and Belonging

Sue Quackenbush

Dapr in Practice thumbnail
24:08
about 2 years ago

Dapr in Practice

marcklefter

Minimising the Carbon Footprint of Workloads thumbnail
27:46
about 2 years ago

Minimising the Carbon Footprint of Workloads

Michael Mueller

Related videos

What If We've Been Scaling Stream Processing Wrong All Along? thumbnail
31:06
World Congress 2026 Europe • 3 months ago

What If We've Been Scaling Stream Processing Wrong All Along?

Hartmut Armbruster

Python-Based Data Streaming Pipelines Within Minutes thumbnail
39:04
LIVE • almost 2 years ago

Python-Based Data Streaming Pipelines Within Minutes

Bobur Umurzokov

Kafka Streams Microservices thumbnail
45:48
World Congress 2021 • about 5 years ago

Kafka Streams Microservices

Denis Washington, Olli Salonen

From event streaming to event sourcing 101 thumbnail
27:33
LIVE • almost 6 years ago

From event streaming to event sourcing 101

Gerard Klijs

Event Messaging and Streaming with Apache Pulsar thumbnail
57:19
LIVE • over 3 years ago

Event Messaging and Streaming with Apache Pulsar

Mary Grygleski

The Rise of Reactive Microservices thumbnail
46:24
World Congress 2022 • over 4 years ago

The Rise of Reactive Microservices

David Leitner

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.