Convert batch code into streaming with Python

What if you could convert your batch pipeline into a real-time stream with just one line of code? This talk introduces the Python framework that makes it possible.

#1about 5 minutes

Why Python is ideal for data streaming frameworks

Python-based frameworks unify streaming and processing components, simplifying connections to data sources and allowing focus on business logic.

#2about 2 minutes

Key use cases for Python streaming frameworks

Explore applications for Python streaming frameworks, including event-driven microservices, real-time data pipelines, and ML/LLM applications.

#3about 2 minutes

Introducing Pathway for unified batch and streaming

Pathway is a Python framework that allows you to build a data pipeline once and run it in both batch and streaming modes with a single configuration change.

#4about 3 minutes

Understanding Pathway's internal data handling and connectors

Data is structured into tables with defined schemas that are automatically updated in real-time, and custom connectors can be built for any data source.

#5about 3 minutes

Building real-time AI applications with Pathway

Use Pathway for real-time data indexing in RAG applications and leverage the llm-app project to avoid vector database synchronization issues.

#6about 7 minutes

Showcasing real-time AI application examples

Review several practical AI applications built with Pathway, including a document Q&A tool, a discount finder, and a real-time alerting system.

#7about 5 minutes

Live demo of a real-time Dropbox Q&A application

A walkthrough of a Python application that connects to Dropbox, indexes documents in real-time, and answers questions across multiple files.

#8about 2 minutes

Key takeaways for modern data processing

Python frameworks offer a unified platform for batch and streaming, enable custom data pipelines, and simplify bringing real-time data to LLM applications.

#9about 6 minutes

Q&A on latency, event processing, and migration challenges

Addressing audience questions about how Pathway ensures low latency, handles complex event processing, and the common challenges of migrating from batch to streaming.

#10about 4 minutes

Q&A on performance, parallelism, and organizational impact

Answering questions about handling data skew, load balancing, data parallelism for speed, and how real-time processing impacts organizational decision-making.

#11about 8 minutes

Q&A on future trends and the developer advocate role

Discussing the future evolution of real-time technologies, resource optimization, UX improvements, and the role of a developer advocate in the tech industry.