WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2024 • Aug 20, 2024 • Session details

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Are you over-engineering your data pipeline? Learn when to leverage complex stream processing frameworks like Apache Flink, and when simple SQL micro-batching is the smarter architectural choice.

Pause
Mute Enter Fullscreen
#1 about 3 min

Defining stream data processing versus standard event processing

Differentiating continuous data aggregation from discrete message decision making.

#2 about 3 min

Handling message order with event and processing time

Analyzing data by actual event occurrence to guarantee offline state consistency.

#3 about 3 min

Ensuring exactly once message delivery in distributed systems

Utilizing stream framework checkpoint mechanisms to completely avoid duplicate data processing.

#4 about 3 min

Constructing data pipelines with stream processing architecture operators

Connecting basic operator functions to correctly read, transform, and sync continuous inputs.

#5 about 6 min

Applying windowing functions to segment unbounded data streams

Segmenting unbounded continuous data streams using distinct tumbling, sliding, and session windows.

#6 about 4 min

Joining data streams and managing stateful memory processing

Combining separate message sources and retaining critical memory state for complex temporal analytics.

#7 about 2 min

Evaluating mature stream processing frameworks for production systems

Reviewing industry-standard data tools like Apache Flink and Beam for scalable application architecture.

#8 about 5 min

Comparing performance and complexity against standard microservice architectures

Benchmarking Apache Beam data pipelines against Spring Boot to assess throughput and setup overhead.

#9 about 3 min

Assessing the organizational cost of stream framework adoption

Weighing high execution speed against steep learning curves and heavy engineering maintenance bandwidth.

#10 about 1 min

Identifying real-world applications for stream processing technology usage

Exploring practical data implementations across gaming anti-cheat systems, telecommunications, and financial fraud detection.

#11 about 2 min

Establishing adoption prerequisites and business stakeholder strategic alignment

Ensuring transparent business requirements exist before committing complex engineering logic to pipeline structural changes.

Matching moments

1:33 min

Overcoming typical barriers to real-time stream processing adoption

Bobur Umurzokov · LIVE

1:56 min

Overview of current stream processing frameworks

Hartmut Armbruster Hartmut Armbruster · WWC Europe 2026

4:14 min

Recognizing architectural drivers pushing event streaming system adoption

Mary Grygleski Mary Grygleski · LIVE

4:30 min

Introducing data management and the shift to streaming

Mary Grygleski Mary Grygleski · LIVE

1:32 min

Comparing stream processing architecture to traditional batch processing

Bobur Umurzokov · LIVE

4:38 min

Operational complexities and performance optimization of stream applications

Denis Washington +1 · WWC 2021

More from World Congress 2024

Beyond the Hype: Real-World AI Strategies Panel thumbnail
24:07
10 months ago

Beyond the Hype: Real-World AI Strategies Panel

Mike Butcher, Jürgen Müller, Katrin Lehmann, Tobias Regenfuss

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation thumbnail
21:17
almost 2 years ago

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation

Carl Lapierre

Coding an Immersive Copilot using Unity / .NET and Azure OpenAI! thumbnail
31:50
almost 2 years ago

Coding an Immersive Copilot using Unity / .NET and Azure OpenAI!

Zaid Zaim

Building a diverse tech company: The power of Diversity, Equity, Inclusion, and Belonging thumbnail
29:43
almost 2 years ago

Building a diverse tech company: The power of Diversity, Equity, Inclusion, and Belonging

Sue Quackenbush

Dapr in Practice thumbnail
24:08
almost 2 years ago

Dapr in Practice

marcklefter

Minimising the Carbon Footprint of Workloads thumbnail
27:46
almost 2 years ago

Minimising the Carbon Footprint of Workloads

Michael Mueller

Upcoming sessions on this topic

Open session

World Congress 2026 North America

AI Agents are Only as Smart as their Context: Building a Real-Time Context Engine at Intuit

Bharat Patel

Lead Software Engineer at Intuit

Bharat Patel
Open session

World Congress 2026 North America

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan
Open session

World Congress 2026 North America

The $0.15/month Algo-Trader: Architecting High-Performance Serverless Trading Engines

Vishwak Thatikonda

Lead Full Stack Engineer at SDH Systems

Vishwak Thatikonda
Open session

World Congress 2026 North America

Ship 10x Faster: AI-Powered Development with Claude Code and MCP Tools

Viktoria Semaan

Principal Technical Evangelist at Databricks

Viktoria Semaan
Open session

World Congress 2026 North America

Offloading IoT State: Solving the Memory Crisis on Legacy Hardware

Pravin Nagare

Senior Software Engineer, Roku

Pravin Nagare

Related videos

What If We've Been Scaling Stream Processing Wrong All Along? thumbnail
31:06
WWC Europe 2026 • about 1 month ago

What If We've Been Scaling Stream Processing Wrong All Along?

Hartmut Armbruster

Python-Based Data Streaming Pipelines Within Minutes thumbnail
39:04
LIVE • almost 2 years ago

Python-Based Data Streaming Pipelines Within Minutes

Bobur Umurzokov

Kafka Streams Microservices thumbnail
45:48
WWC 2021 • about 5 years ago

Kafka Streams Microservices

Denis Washington, Olli Salonen

From event streaming to event sourcing 101 thumbnail
27:33
LIVE • over 5 years ago

From event streaming to event sourcing 101

Gerard Klijs

Event Messaging and Streaming with Apache Pulsar thumbnail
57:19
LIVE • over 3 years ago

Event Messaging and Streaming with Apache Pulsar

Mary Grygleski

The Rise of Reactive Microservices thumbnail
46:24
WWC 2022 • about 4 years ago

The Rise of Reactive Microservices

David Leitner

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.