WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Oct 12, 2020

PySpark - Combining Machine Learning & Big Data

Ayon Roy

Bypass complex Scala programming and deploy native Python operations across distributed clusters. Learn how PySpark and MLlib unite to accelerate robust, scalable machine learning pipelines.

Pause
Mute Enter Fullscreen
#1 about 4 min

Defining big data and machine learning fundamentals

The four Vs conceptualize big data while machine learning enables systems to predict outcomes without explicit programming.

#2 about 3 min

Why organizations combine big data and machine learning

Tapping into massive amounts of new data with machine learning drives personalized consumer experiences.

#3 about 3 min

Capabilities of the Apache Spark processing engine

Spark provides a large-scale data processing environment with integrated libraries for machine learning, SQL, and streaming.

#4 about 4 min

Understanding RDDs, DataFrames, and Datasets in Spark

Data ingestion evolved from complex resilient distributed datasets into developer-friendly structures like DataFrames and Datasets.

#5 about 5 min

Exploring the Apache Spark layer architecture

Cluster managers drive resource allocation while Spark Core processes diverse data sources using top-layer libraries.

#6 about 8 min

How cluster managers orchestrate distributed tasks

A driver program coordinates with a cluster manager via a Spark context to execute computations in parallel on worker nodes.

#7 about 5 min

Harnessing Spark with Python using PySpark and Py4J

The PySpark library relies on Py4J sockets to translate Python instructions into processes compatible with Java Virtual Machines.

#8 about 5 min

Advantages of utilizing Spark's native MLlib

Built-in classification, feature extraction, and pipeline support make MLlib a scalable choice for data analytics.

#9 about 5 min

Constructing ML pipelines with transformers and estimators

DataFrames pass through transformers to engineer features and enter estimators that output functional predictive models.

#10 about 4 min

Pre-built algorithms and further learning resources

Spark MLlib offers accessible, pre-built regression and clustering algorithms for integrating big data processing with predictive modeling.

Matching moments

2:56 min

Options for database machine learning integration architectures

Akmal Chaudhri Akmal Chaudhri · LIVE

4:18 min

Exploring the big data and machine learning portfolio

Qiyang Duan · LIVE

2:04 min

Comparing offline data analytics with online stream processing

Artem Volk Artem Volk +1 · WWC 2024

1:41 min

Visualizing the complex developer journey for JVM ecosystems

Bobur Umurzokov · LIVE

15:08 min

Audience questions on practical machine learning operational strategies

Lina Weichbrodt · LIVE

7:10 min

Exploring pathways into the machine learning engineering field

Jose Luis Latorre Millas · LIVE

More from WeAreDevelopers LIVE

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
5 days ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Node and Package Security thumbnail
1:08:30
5 days ago

WeAreDevelopers LIVE - Node and Package Security

Chris Heilmann, Daniel Cranney, Zbyszek Tenerowicz

WeAreDevelopers LIVE - Streaming HTML thumbnail
1:10:26
12 days ago

WeAreDevelopers LIVE - Streaming HTML

Chris Heilmann, Daniel Cranney, Jad Joubran

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe thumbnail
1:02:34
19 days ago

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe

Chris Heilmann, Daniel Cranney, Justin Halsall, Werner Jeschko, Nicole Jeschko, Mirco Tomić

WeAreDevelopers LIVE - Web Performance in an AI World thumbnail
1:07:59
26 days ago

WeAreDevelopers LIVE - Web Performance in an AI World

Chris Heilmann, Daniel Cranney

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko thumbnail
5:55
about 2 months ago

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

World Congress 2026 North America

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan
Open session

World Congress 2026 North America

Ship 10x Faster: AI-Powered Development with Claude Code and MCP Tools

Viktoria Semaan

Principal Technical Evangelist at Databricks

Viktoria Semaan
Open session

World Congress 2026 North America

Look What Java Can Do Now: Live-Coding a GenAI MCP Server with the JAQ Stack

Suren Konathala

Senior Engineering Leader, Marketing & AdTech, AI GTM & Digital Experience Platforms, Java x AI

Suren Konathala
Open session

World Congress 2026 North America

Building the AI Era Through Developer Communities: A Competitive Advantage for Engineers & Teams

John Komarnicki

National Executive Advisor at Code & Coffee | Engineering & AI Community Leader

John Komarnicki
Open session

World Congress 2026 North America

Beyond the Code: Human-AI Synergies in Product Development

Ajita Kanchivakam Ananth

Staff Technical Program Manager at Google

Ajita Kanchivakam Ananth
Open session

World Congress 2026 North America

Building Pragmatic AI: 10 AI Features Your Users Actually Want

Jonathan "J." Tower

.NET Foundation Board | 12x Microsoft MVP | Founder & Consultant

Jonathan "J." Tower

Related videos

Alibaba Big Data and Machine Learning Technology thumbnail
43:57
LIVE • almost 6 years ago

Alibaba Big Data and Machine Learning Technology

Qiyang Duan

Overview of Machine Learning in Python thumbnail
57:46
LIVE • over 2 years ago

Overview of Machine Learning in Python

Adrian Schmitt

Introduction to Azure Machine Learning thumbnail
54:27
LIVE • over 4 years ago

Introduction to Azure Machine Learning

Jose Luis Latorre Millas

Data Science in Retail thumbnail
58:21
LIVE • about 3 years ago

Data Science in Retail

Julian Joseph

Let's Get Aggregated: Custom UDAFs in Spark thumbnail
19:58
WWC 2025 • 12 months ago

Let's Get Aggregated: Custom UDAFs in Spark

Nikolai Nikolaev

Data Fabric in Action - How to enhance a Stock Trading App with ML and Data Virtualization thumbnail
45:58
LIVE • almost 5 years ago

Data Fabric in Action - How to enhance a Stock Trading App with ML and Data Virtualization

Andreas Christian

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.