WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Oct 30, 2024

Python-Based Data Streaming Pipelines Within Minutes

Bobur Umurzokov

Tired of wrestling with heavy JVM ecosystems for real-time data streaming? Discover how to deploy serverless, AI-enriched pipelines in minutes using pure Python.

Pause
Mute Enter Fullscreen
#1 about 3 min

Rise of Python in real-time data processing

The shift toward natively using Python frameworks instead of Java wrappers for stream processing applications.

#2 about 2 min

Overcoming typical barriers to real-time stream processing adoption

Why high infrastructure investments and operational mental shifts often hesitate companies from moving beyond batch processing.

#3 about 2 min

Designing a real-time price recommendation system for vehicles

An engineering scenario detailing how incoming vehicle registrations require continuous price estimations via language models.

#4 about 2 min

Comparing stream processing architecture to traditional batch processing

How real-time transformation layers constantly aggregate and enrich data directly from source to destination.

#5 about 2 min

Building an initial solution using Debezium and Apache Kafka

Attempting to capture database changes and pass messaging events into a primary AI-driven Python service.

#6 about 4 min

Infrastructure challenges when combining Kafka with Apache Flink

The complications of orchestrating massive compute and messaging layers requiring dedicated JVM engineers.

#7 about 3 min

Hidden costs of self-hosting and managed Kafka solutions

How maintenance updates, performance monitoring, and over-provisioning create massive overhead for isolated data teams.

#8 about 2 min

Visualizing the complex developer journey for JVM ecosystems

The steep learning curve involving external state stores, Spark deployment, and Databricks just to process basic events.

#9 about 2 min

Unifying stream processing workflows natively within Python frameworks

Removing operational friction by merging the broker and processor layers into simplified Python-based environments.

#10 about 3 min

Constructing serverless streaming pipelines using GlassFlow infrastructure

Demonstrating a low-code approach that auto-scales internal messaging mechanisms across simple Python functions.

#11 about 4 min

Real-world scenarios for clickstream analytics and anomaly detection

Practical integration of continuous data streams into vector databases, live Plotly dashboards, and Slack notifications.

#12 about 5 min

Provisioning a direct GlassFlow pipeline connected to Supabase

Initializing a real-time table structure and preparing a webhook destination to capture persistent updates.

#13 about 4 min

Configuring Python transformation functions for price predictions

Creating logic that passes incoming dictionary formats from the source system into AI dependencies for validation.

#14 about 2 min

Synchronizing database webhook triggers with pipeline webhooks

Safely binding source credentials so that every table insertion notifies the stream processor without delays.

#15 about 3 min

Validating end-to-end functionality using direct SQL insertions

Confirming immediate remote prediction delivery upon raw query executions against the initialized database container.

#16 about 4 min

Summarizing Python frameworks advantages and future event streams

Reflections on overcoming architectural silos and encouraging participation in deeper design dialogues around event-driven paradigms.

Matching moments

2:04 min

Comparing offline data analytics with online stream processing

Artem Volk Artem Volk +1 · World Congress 2024

4:14 min

Recognizing architectural drivers pushing event streaming system adoption

Mary Grygleski Mary Grygleski · LIVE

3:33 min

Refactoring data science workflows using Rapids QDF and Pandas

Paul Graham Paul Graham · LIVE

1:09 min

Evaluating mature stream processing frameworks for production systems

Soroosh Khodami Soroosh Khodami · World Congress 2024

1:56 min

Overview of current stream processing frameworks

Hartmut Armbruster Hartmut Armbruster · World Congress 2026 Europe

5:15 min

Leveraging server-side streaming for performance and interactivity

Chris Heilmann Chris Heilmann +2 · LIVE

More from WeAreDevelopers LIVE

WeAreDevelopers LIVE - Is CSS Catching Up to Flash? thumbnail
1:03:33
10 days ago

WeAreDevelopers LIVE - Is CSS Catching Up to Flash?

Chris Heilmann, Daniel Cranney, Bramus Van Damme

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along thumbnail
1:21:17
17 days ago

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along

Chris Heilmann, Daniel Cranney, Krzysztof Cieślak

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak thumbnail
4:23
19 days ago

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Open Source Agent Toolkits thumbnail
1:00:38
24 days ago

WeAreDevelopers LIVE - Open Source Agent Toolkits

Chris Heilmann, Daniel Cranney, Eddie Jaoude

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude thumbnail
6:16
26 days ago

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude

Chris Heilmann, Daniel Cranney

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail
5:46
about 1 month ago

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

Related videos

Why and when should we consider Stream Processing frameworks in our solutions thumbnail
30:51
World Congress 2024 • about 2 years ago

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Let's Get Started With Apache Kafka® for Python Developers thumbnail
38:50
LIVE • over 3 years ago

Let's Get Started With Apache Kafka® for Python Developers

Lucia Cerchie

Python Data Visualization @ Deepnote (w/ PyViz overview) thumbnail
50:38
LIVE • over 5 years ago

Python Data Visualization @ Deepnote (w/ PyViz overview)

Radovan Kavický

What If We've Been Scaling Stream Processing Wrong All Along? thumbnail
31:06
World Congress 2026 Europe • 3 months ago

What If We've Been Scaling Stream Processing Wrong All Along?

Hartmut Armbruster

Implementing continuous delivery in a data processing pipeline thumbnail
34:00
LIVE • almost 6 years ago

Implementing continuous delivery in a data processing pipeline

Álvaro Martín Lozano

Tips, Techniques, and Common Pitfalls Debugging Kafka thumbnail
54:29
LIVE • almost 3 years ago

Tips, Techniques, and Common Pitfalls Debugging Kafka

Developersteve

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.