WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Feb 23, 2024

Convert batch code into streaming with Python

Bobur Umurzokov

What if you could switch from static batch data to live streaming with one configuration change? Learn to ditch complex JVMs and power real-time AI applications using pure Python.

Pause
Mute Enter Fullscreen
#1 about 6 min

Advantages of Python frameworks for data streaming

An introduction to Python's capability for unifying stream processing and data platforms without complex management layers.

#2 about 2 min

Use cases for stream processing with Python frameworks

Python stream processing complements microservices architectures and enables real-time vector embeddings for machine learning.

#3 about 6 min

Building unified processing pipelines with the Pathway framework

Pathway provides a single python code base to smoothly transition from testing on static data to running streaming inputs.

#4 about 4 min

Integrating real-time data indexing into AI applications

Real-time continuous updates generate vector embeddings for augmented generation without requiring separate vector storage synchronizations.

#5 about 4 min

Examples of real-time artificial intelligence application development

Practical applications include summarizing unstructured documents, identifying real-time market discounts, and generating automated alerts.

#6 about 4 min

Code and architecture for building the discount application

A walkthrough of a python backend that extracts data and utilizes user interfaces via streamlit and openai endpoints.

#7 about 6 min

Demonstrating a real-time Dropbox document summarization application

A dockerized application chunks localized text and continuously polls for query responses to summarize expense invoices.

#8 about 2 min

Final takeaways on Python for streaming and batch processing

Python frameworks simplify the transition to streaming workloads by natively compiling python logic without relying on java abstractions.

#9 about 2 min

Understanding how Pathway ensures low latency data processing

Dynamic detection of newly added table row streams enables rapid state updates directly within memory-based inputs.

#10 about 2 min

Handling complex event processing and pattern recognition formats

Kafka event brokers seamlessly ingest dynamic column adjustments within tables without entirely dropping schema configurations.

#11 about 2 min

Limitations when migrating batch processes to native streaming

Migrating to streaming infrastructures introduces common limitations regarding learning curves, connector incompatibility, and complete pipeline rebuilds.

#12 about 1 min

Managing automatic load balancing and data skew effectively

Managed services balance processing loads by splitting transformations across instances and continuously merging output states.

#13 about 2 min

Implementing data parallelism constraints to improve processing speeds

Breaking complex transformation logic into smaller steps guarantees optimal scaling mechanisms without blocking sequential dependencies.

#14 about 2 min

Impact of real-time processing on organizational decision making

Continually cycling analytics enables faster customer experiences by prioritizing fresh data visualizations over overnight batch delays.

#15 about 3 min

Future evolution of real-time data processing technologies

Startups are transitioning storage tools into real-time streaming databases that combine unified architectures natively.

#16 about 2 min

Resource utilization techniques in real-time processing environments

Restricting continuous memory operations eliminates the excessive data cluster expansion and storage distributions required by rigid frameworks.

#17 about 2 min

Improving user experiences using directly connected real-time sources

Removing backend logic layers empowers frontline components to autonomously generate dynamic visualizations driven by real-time streams.

#18 about 2 min

Defining the core responsibilities of a developer advocate

Developer advocates leverage their technical software backgrounds to effectively communicate product benefits directly with engineering teams.

Matching moments

2:21 min

Rise of Python in real-time data processing

Bobur Umurzokov · LIVE

1:33 min

Overcoming typical barriers to real-time stream processing adoption

Bobur Umurzokov · LIVE

4:30 min

Introducing data management and the shift to streaming

Mary Grygleski Mary Grygleski · LIVE

1:32 min

Comparing stream processing architecture to traditional batch processing

Bobur Umurzokov · LIVE

4:14 min

Recognizing architectural drivers pushing event streaming system adoption

Mary Grygleski Mary Grygleski · LIVE

3:09 min

Summarizing Python frameworks advantages and future event streams

Bobur Umurzokov · LIVE

More from WeAreDevelopers LIVE

Fake or News - Claude's DNA Discovery, Quick Cloning & Crime on Insta thumbnail
6:07
1 day ago

Fake or News - Claude's DNA Discovery, Quick Cloning & Crime on Insta

Andrew MacLean, Daniel Cranney, Christian Heilmann

WeAreDevelopers LIVE - Playing in Production thumbnail
1:06:40
1 day ago

WeAreDevelopers LIVE - Playing in Production

Andrew MacLean, Daniel Cranney, Christian Heilmann

WeAreDevelopers LIVE - Is CSS Catching Up to Flash? thumbnail
1:03:33
15 days ago

WeAreDevelopers LIVE - Is CSS Catching Up to Flash?

Chris Heilmann, Daniel Cranney, Bramus Van Damme

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along thumbnail
1:21:17
22 days ago

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along

Chris Heilmann, Daniel Cranney, Krzysztof Cieślak

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak thumbnail
4:23
24 days ago

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Open Source Agent Toolkits thumbnail
1:00:38
29 days ago

WeAreDevelopers LIVE - Open Source Agent Toolkits

Chris Heilmann, Daniel Cranney, Eddie Jaoude

Related videos

Python-Based Data Streaming Pipelines Within Minutes thumbnail
39:04
LIVE • almost 2 years ago

Python-Based Data Streaming Pipelines Within Minutes

Bobur Umurzokov

Why and when should we consider Stream Processing frameworks in our solutions thumbnail
30:51
World Congress 2024 • about 2 years ago

Why and when should we consider Stream Processing frameworks in our solutions

Soroosh Khodami

Let's Get Started With Apache Kafka® for Python Developers thumbnail
38:50
LIVE • over 3 years ago

Let's Get Started With Apache Kafka® for Python Developers

Lucia Cerchie

What If We've Been Scaling Stream Processing Wrong All Along? thumbnail
31:06
World Congress 2026 Europe • 3 months ago

What If We've Been Scaling Stream Processing Wrong All Along?

Hartmut Armbruster

Implementing continuous delivery in a data processing pipeline thumbnail
34:00
LIVE • almost 6 years ago

Implementing continuous delivery in a data processing pipeline

Álvaro Martín Lozano

Kafka Streams Microservices thumbnail
45:48
World Congress 2021 • over 5 years ago

Kafka Streams Microservices

Denis Washington, Olli Salonen

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.