> Markdown version of [/playlists/data-pipelines](https://www.wearedevelopers.com/playlists/data-pipelines). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Playlist: Data pipelines 16 videos · 16 moments · 61.4 minutes ## Data Analytics with Microsoft Fabric: End-to-End Use Case with Data Agents - **Building modern data pipelines for legacy exports** (11:12, 0min) — Translating bulky legacy customer system exports seamlessly into high-performance Delta tables for scalable analytic pipelines. [Learn more](https://www.wearedevelopers.com/videos/1547-data-analytics-with-microsoft-fabric-end-to-end-use-case-with-data-agents) ## Modern Data Architectures need Software Engineering - **Applying software engineering environments and testing to data pipelines** (08:36, 6min) — Implementing unit testing and proper multi-environment deployment structures ensures robust data pipelines in production. [Learn more](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) ## Hacking Your Vacation: Using Data for Fun - **Extrapolating data collection methods and cloud architecture data pipelines** (49:03, 8min) — Clarifying how wait times are continuously scraped and stored directly into cloud databases for ongoing analysis. [Learn more](https://www.wearedevelopers.com/videos/585-hacking-your-vacation-using-data-for-fun) ## Why and when should we consider Stream Processing frameworks in our solutions - **Constructing data pipelines with stream processing architecture operators** (08:16, 2min) — Connecting basic operator functions to correctly read, transform, and sync continuous inputs. [Learn more](https://www.wearedevelopers.com/videos/1085-why-and-when-should-we-consider-stream-processing-frameworks-in-our-solutions) ## Harry Potter and the Elastic Semantic Search - **Automating data pipeline updates for production environments** (44:00, 1min) — Configuring continual ingestion workflows prevents indexing disruptions across dynamic document infrastructures. [Learn more](https://www.wearedevelopers.com/videos/860-harry-potter-and-the-elastic-semantic-search) ## DevOps for Machine Learning - **Building data pipelines and managing machine learning features** (10:26, 3min) — Centralized feature stores transform raw data into reusable and versioned elements for multiple project teams. [Learn more](https://www.wearedevelopers.com/videos/179-devops-for-machine-learning) ## Data Science in Retail - **Scaling machine learning pipelines from prototypes to petabytes** (31:36, 3min) — Structuring robust data engineering workflows addresses the transition from subset analysis to large-scale production deployments. [Learn more](https://www.wearedevelopers.com/videos/586-data-science-in-retail) ## Event Messaging and Streaming with Apache Pulsar - **Transforming data pipelines natively through lightweight serverless functions** (39:26, 3min) — Enriching raw data dynamically using integrated schema registries and versatile input-output platform connectors. [Learn more](https://www.wearedevelopers.com/videos/538-event-messaging-and-streaming-with-apache-pulsar) ## Implementing continuous delivery in a data processing pipeline - **Implementing continuous deployment architectures for data pipelines** (11:34, 2min) — Constructing pipelines utilizing isolated local testing, dedicated staging generation, and explicit smoke tests. [Learn more](https://www.wearedevelopers.com/videos/73-implementing-continuous-delivery-in-a-data-processing-pipeline) ## How Gatsby Cloud's real-time streaming architecture drives <5 second builds - **Using database indexes and reactive processing patterns** (18:35, 5min) — How explicit data pipelines use targeted transformations to react immediately to single input updates. [Learn more](https://www.wearedevelopers.com/videos/418-how-gatsby-cloud-s-real-time-streaming-architecture-drives-5-second-builds) ## Agentic AI in Go - **Composing autonomous data pipelines across multiple models** (22:19, 3min) — Orchestrating distinct specialized models to decompose complex spreadsheet forms into actionable and programmatic data structures. [Learn more](https://www.wearedevelopers.com/videos/100274-agentic-ai-in-go) ## Kafka Streams Microservices - **Building a consistent product catalog stream data pipeline** (13:14, 3min) — Aggregating, cleaning, and enriching unreliable inbound event streams into an interactive full-text search view. [Learn more](https://www.wearedevelopers.com/videos/168-kafka-streams-microservices) ## Building Blocks of RAG: From Understanding to Implementation - **Visualizing the end-to-end data pipeline and application workflow** (17:09, 2min) — A step-by-step architectural breakdown visualizes data ingestion, vector storage, query embedding, and final response generation. [Learn more](https://www.wearedevelopers.com/videos/1249-building-blocks-of-rag-from-understanding-to-implementation) ## Introduction to Azure Machine Learning - **Visualizing data pipelines using Azure Machine Learning Designer** (19:05, 6min) — Utilizing an interactive drag-and-drop workspace simplifies the visual validation of normalization functions and dataset splittings. [Learn more](https://www.wearedevelopers.com/videos/368-introduction-to-azure-machine-learning) ## 30 Golden Rules of Deep Learning Performance - **Building asynchronous data pipelines with TensorFlow data APIs** (16:46, 4min) — Interleaving batch fetching inside background threads ensures the active GPU is constantly being fed data arrays. [Learn more](https://www.wearedevelopers.com/videos/11-30-golden-rules-of-deep-learning-performance) ## Parquet, Delta, Iceberg & Ducklake - An introduction for developers - **Introduction to analytical data formats for software developers** (00:00, 0min) — Why software engineers need to understand the underlying infrastructure of data engineering pipelines. [Learn more](https://www.wearedevelopers.com/videos/100075-parquet-delta-iceberg-ducklake-an-introduction-for-developers)