WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Oct 30, 2024

Python-Based Data Streaming Pipelines Within Minutes

Bobur Umurzokov

Tired of wrestling with heavy JVM ecosystems for real-time data streaming? Discover how to deploy serverless, AI-enriched pipelines in minutes using pure Python.

Pause
Mute Enter Fullscreen
#1 about 3 min

Rise of Python in real-time data processing

The shift toward natively using Python frameworks instead of Java wrappers for stream processing applications.

#2 about 2 min

Overcoming typical barriers to real-time stream processing adoption

Why high infrastructure investments and operational mental shifts often hesitate companies from moving beyond batch processing.

#3 about 2 min

Designing a real-time price recommendation system for vehicles

An engineering scenario detailing how incoming vehicle registrations require continuous price estimations via language models.

#4 about 2 min

Comparing stream processing architecture to traditional batch processing

How real-time transformation layers constantly aggregate and enrich data directly from source to destination.

#5 about 2 min

Building an initial solution using Debezium and Apache Kafka

Attempting to capture database changes and pass messaging events into a primary AI-driven Python service.

#6 about 4 min

Infrastructure challenges when combining Kafka with Apache Flink

The complications of orchestrating massive compute and messaging layers requiring dedicated JVM engineers.

#7 about 3 min

Hidden costs of self-hosting and managed Kafka solutions

How maintenance updates, performance monitoring, and over-provisioning create massive overhead for isolated data teams.

#8 about 2 min

Visualizing the complex developer journey for JVM ecosystems

The steep learning curve involving external state stores, Spark deployment, and Databricks just to process basic events.

#9 about 2 min

Unifying stream processing workflows natively within Python frameworks

Removing operational friction by merging the broker and processor layers into simplified Python-based environments.

#10 about 3 min

Constructing serverless streaming pipelines using GlassFlow infrastructure

Demonstrating a low-code approach that auto-scales internal messaging mechanisms across simple Python functions.

#11 about 4 min

Real-world scenarios for clickstream analytics and anomaly detection

Practical integration of continuous data streams into vector databases, live Plotly dashboards, and Slack notifications.

#12 about 5 min

Provisioning a direct GlassFlow pipeline connected to Supabase

Initializing a real-time table structure and preparing a webhook destination to capture persistent updates.

#13 about 4 min

Configuring Python transformation functions for price predictions

Creating logic that passes incoming dictionary formats from the source system into AI dependencies for validation.

#14 about 2 min

Synchronizing database webhook triggers with pipeline webhooks

Safely binding source credentials so that every table insertion notifies the stream processor without delays.

#15 about 3 min

Validating end-to-end functionality using direct SQL insertions

Confirming immediate remote prediction delivery upon raw query executions against the initialized database container.

#16 about 4 min

Summarizing Python frameworks advantages and future event streams

Reflections on overcoming architectural silos and encouraging participation in deeper design dialogues around event-driven paradigms.

Matching moments

2:04 min

Comparing offline data analytics with online stream processing

Artem Volk Artem Volk +1 · WWC 2024

4:14 min

Recognizing architectural drivers pushing event streaming system adoption

Mary Grygleski Mary Grygleski · LIVE

3:33 min

Refactoring data science workflows using Rapids QDF and Pandas

Paul Graham Paul Graham · LIVE

1:09 min

Evaluating mature stream processing frameworks for production systems

Soroosh Khodami Soroosh Khodami · WWC 2024

1:56 min

Overview of current stream processing frameworks

Hartmut Armbruster Hartmut Armbruster · WWC Europe 2026

5:15 min

Leveraging server-side streaming for performance and interactivity

Chris Heilmann +2 · LIVE

More from WeAreDevelopers LIVE

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
5 days ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Node and Package Security thumbnail
1:08:30
5 days ago

WeAreDevelopers LIVE - Node and Package Security

Chris Heilmann, Daniel Cranney, Zbyszek Tenerowicz

WeAreDevelopers LIVE - Streaming HTML thumbnail
1:10:26
12 days ago

WeAreDevelopers LIVE - Streaming HTML

Chris Heilmann, Daniel Cranney, Jad Joubran

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe thumbnail
1:02:34
19 days ago

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe

Chris Heilmann, Daniel Cranney, Justin Halsall, Werner Jeschko, Nicole Jeschko, Mirco Tomić

WeAreDevelopers LIVE - Web Performance in an AI World thumbnail
1:07:59
26 days ago

WeAreDevelopers LIVE - Web Performance in an AI World

Chris Heilmann, Daniel Cranney

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko thumbnail
5:55
about 2 months ago

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

World Congress 2026 North America

AI Agents are Only as Smart as their Context: Building a Real-Time Context Engine at Intuit

Bharat Patel

Lead Software Engineer at Intuit

Bharat Patel
Open session

World Congress 2026 North America

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

The $0.15/month Algo-Trader: Architecting High-Performance Serverless Trading Engines

Vishwak Thatikonda

Lead Full Stack Engineer at SDH Systems

Vishwak Thatikonda
Open session

World Congress 2026 North America

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan
Open session

World Congress 2026 North America

Agents That Own Their Inference: Building Production AI Agents on Dedicated GPUs

Duan Lightfoot

Sr. AI Engineer, Akamai

Duan Lightfoot
Open session

World Congress 2026 North America

The Broken Rung: How AI is Rebuilding Software Development from the Ground Up

Tomislav Tipurić

Chief Technology Officer, Nephos

Tomislav Tipurić

Related videos

Why and when should we consider Stream Processing frameworks in our solutions thumbnail
30:51
WWC 2024 • almost 2 years ago

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Let's Get Started With Apache Kafka® for Python Developers thumbnail
38:50
LIVE • over 3 years ago

Let's Get Started With Apache Kafka® for Python Developers

Lucia Cerchie

Python Data Visualization @ Deepnote (w/ PyViz overview) thumbnail
50:38
LIVE • over 5 years ago

Python Data Visualization @ Deepnote (w/ PyViz overview)

Radovan Kavický

What If We've Been Scaling Stream Processing Wrong All Along? thumbnail
31:06
WWC Europe 2026 • about 1 month ago

What If We've Been Scaling Stream Processing Wrong All Along?

Hartmut Armbruster

Implementing continuous delivery in a data processing pipeline thumbnail
34:00
LIVE • over 5 years ago

Implementing continuous delivery in a data processing pipeline

Álvaro Martín Lozano

Tips, Techniques, and Common Pitfalls Debugging Kafka thumbnail
54:29
LIVE • over 2 years ago

Tips, Techniques, and Common Pitfalls Debugging Kafka

Developersteve

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.