WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Oct 30, 2024

Python-Based Data Streaming Pipelines Within Minutes

Bobur Umurzokov

Tired of wrestling with heavy JVM ecosystems for real-time data streaming? Discover how to deploy serverless, AI-enriched pipelines in minutes using pure Python.

Pause
Mute Enter Fullscreen
#1 about 3 min

Rise of Python in real-time data processing

The shift toward natively using Python frameworks instead of Java wrappers for stream processing applications.

#2 about 2 min

Overcoming typical barriers to real-time stream processing adoption

Why high infrastructure investments and operational mental shifts often hesitate companies from moving beyond batch processing.

#3 about 2 min

Designing a real-time price recommendation system for vehicles

An engineering scenario detailing how incoming vehicle registrations require continuous price estimations via language models.

#4 about 2 min

Comparing stream processing architecture to traditional batch processing

How real-time transformation layers constantly aggregate and enrich data directly from source to destination.

#5 about 2 min

Building an initial solution using Debezium and Apache Kafka

Attempting to capture database changes and pass messaging events into a primary AI-driven Python service.

#6 about 4 min

Infrastructure challenges when combining Kafka with Apache Flink

The complications of orchestrating massive compute and messaging layers requiring dedicated JVM engineers.

#7 about 3 min

Hidden costs of self-hosting and managed Kafka solutions

How maintenance updates, performance monitoring, and over-provisioning create massive overhead for isolated data teams.

#8 about 2 min

Visualizing the complex developer journey for JVM ecosystems

The steep learning curve involving external state stores, Spark deployment, and Databricks just to process basic events.

#9 about 2 min

Unifying stream processing workflows natively within Python frameworks

Removing operational friction by merging the broker and processor layers into simplified Python-based environments.

#10 about 3 min

Constructing serverless streaming pipelines using GlassFlow infrastructure

Demonstrating a low-code approach that auto-scales internal messaging mechanisms across simple Python functions.

#11 about 4 min

Real-world scenarios for clickstream analytics and anomaly detection

Practical integration of continuous data streams into vector databases, live Plotly dashboards, and Slack notifications.

#12 about 5 min

Provisioning a direct GlassFlow pipeline connected to Supabase

Initializing a real-time table structure and preparing a webhook destination to capture persistent updates.

#13 about 4 min

Configuring Python transformation functions for price predictions

Creating logic that passes incoming dictionary formats from the source system into AI dependencies for validation.

#14 about 2 min

Synchronizing database webhook triggers with pipeline webhooks

Safely binding source credentials so that every table insertion notifies the stream processor without delays.

#15 about 3 min

Validating end-to-end functionality using direct SQL insertions

Confirming immediate remote prediction delivery upon raw query executions against the initialized database container.

#16 about 4 min

Summarizing Python frameworks advantages and future event streams

Reflections on overcoming architectural silos and encouraging participation in deeper design dialogues around event-driven paradigms.

Matching moments

2:04 min

Comparing offline data analytics with online stream processing

Artem Volk Artem Volk +1 · World Congress 2024

4:14 min

Recognizing architectural drivers pushing event streaming system adoption

Mary Grygleski Mary Grygleski · LIVE

3:33 min

Refactoring data science workflows using Rapids QDF and Pandas

Paul Graham Paul Graham · LIVE

1:09 min

Evaluating mature stream processing frameworks for production systems

Soroosh Khodami Soroosh Khodami · World Congress 2024

1:56 min

Overview of current stream processing frameworks

Hartmut Armbruster Hartmut Armbruster · World Congress 2026 Europe

5:15 min

Leveraging server-side streaming for performance and interactivity

Chris Heilmann +2 · LIVE

More from WeAreDevelopers LIVE

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude thumbnail
6:16
9 days ago

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude

Chris Heilmann, Daniel Cranney

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail
5:46
14 days ago

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

WeAreDevelopers - Of Agents and Robots thumbnail
1:04:42
14 days ago

WeAreDevelopers - Of Agents and Robots

Chris Heilmann, Daniel Cranney, Marc Plogas

WeAreDevelopers LIVE - How DevRel Makes Tech More Human thumbnail
48:06
21 days ago

WeAreDevelopers LIVE - How DevRel Makes Tech More Human

Chris Heilmann, Daniel Cranney

Fake or News: Smart Glasses in Court, Medical Advice Watches & Mushroom Electonics - Jason Torres thumbnail
7:41
22 days ago

Fake or News: Smart Glasses in Court, Medical Advice Watches & Mushroom Electonics - Jason Torres

Chris Heilmann, Daniel Cranney

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
about 1 month ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 25, 2026 · 09:00–09:30

Stage 7

AI Agents are Only as Smart as their Context: Building a Real-Time Context Engine at Intuit

Bharat Patel

Lead Software Engineer at Intuit

Bharat Patel
Open session

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 3

Real-Time Data Platforms at Trillion-Event Scale

Diptamay Sanyal

Principal Engineer | Data, AI & Cybersecurity Platforms

Diptamay Sanyal
Open session

World Congress 2026 North America

September 24, 2026 · 16:50–17:20

Stage 9

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

September 25, 2026 · 14:50–15:20

Stage 7

The $0.15/month Algo-Trader: Architecting High-Performance Serverless Trading Engines

Vishwak Thatikonda

Lead Full Stack Engineer at SDH Systems

Vishwak Thatikonda
Open session

World Congress 2026 North America

September 25, 2026 · 11:40–12:10

Stage 9

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan
Open session

World Congress 2026 North America

September 24, 2026 · 11:00–11:30

Stage 1

Application-Defined Compute: Rethinking Infrastructure for AI Applications

Anurag Goel

Founder & CEO of Render

Anurag Goel

Related videos

Why and when should we consider Stream Processing frameworks in our solutions thumbnail
30:51
World Congress 2024 • about 2 years ago

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Let's Get Started With Apache Kafka® for Python Developers thumbnail
38:50
LIVE • over 3 years ago

Let's Get Started With Apache Kafka® for Python Developers

Lucia Cerchie

Python Data Visualization @ Deepnote (w/ PyViz overview) thumbnail
50:38
LIVE • over 5 years ago

Python Data Visualization @ Deepnote (w/ PyViz overview)

Radovan Kavický

What If We've Been Scaling Stream Processing Wrong All Along? thumbnail
31:06
World Congress 2026 Europe • 2 months ago

What If We've Been Scaling Stream Processing Wrong All Along?

Hartmut Armbruster

Implementing continuous delivery in a data processing pipeline thumbnail
34:00
LIVE • almost 6 years ago

Implementing continuous delivery in a data processing pipeline

Álvaro Martín Lozano

Tips, Techniques, and Common Pitfalls Debugging Kafka thumbnail
54:29
LIVE • over 2 years ago

Tips, Techniques, and Common Pitfalls Debugging Kafka

Developersteve

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.