WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Oct 12, 2020

PySpark - Combining Machine Learning & Big Data

Ayon Roy

Bypass complex Scala programming and deploy native Python operations across distributed clusters. Learn how PySpark and MLlib unite to accelerate robust, scalable machine learning pipelines.

Pause
Mute Enter Fullscreen
#1 about 4 min

Defining big data and machine learning fundamentals

The four Vs conceptualize big data while machine learning enables systems to predict outcomes without explicit programming.

#2 about 3 min

Why organizations combine big data and machine learning

Tapping into massive amounts of new data with machine learning drives personalized consumer experiences.

#3 about 3 min

Capabilities of the Apache Spark processing engine

Spark provides a large-scale data processing environment with integrated libraries for machine learning, SQL, and streaming.

#4 about 4 min

Understanding RDDs, DataFrames, and Datasets in Spark

Data ingestion evolved from complex resilient distributed datasets into developer-friendly structures like DataFrames and Datasets.

#5 about 5 min

Exploring the Apache Spark layer architecture

Cluster managers drive resource allocation while Spark Core processes diverse data sources using top-layer libraries.

#6 about 8 min

How cluster managers orchestrate distributed tasks

A driver program coordinates with a cluster manager via a Spark context to execute computations in parallel on worker nodes.

#7 about 5 min

Harnessing Spark with Python using PySpark and Py4J

The PySpark library relies on Py4J sockets to translate Python instructions into processes compatible with Java Virtual Machines.

#8 about 5 min

Advantages of utilizing Spark's native MLlib

Built-in classification, feature extraction, and pipeline support make MLlib a scalable choice for data analytics.

#9 about 5 min

Constructing ML pipelines with transformers and estimators

DataFrames pass through transformers to engineer features and enter estimators that output functional predictive models.

#10 about 4 min

Pre-built algorithms and further learning resources

Spark MLlib offers accessible, pre-built regression and clustering algorithms for integrating big data processing with predictive modeling.

Matching moments

2:56 min

Options for database machine learning integration architectures

Akmal Chaudhri Akmal Chaudhri · LIVE

4:18 min

Exploring the big data and machine learning portfolio

Qiyang Duan · LIVE

2:04 min

Comparing offline data analytics with online stream processing

Artem Volk Artem Volk +1 · World Congress 2024

1:41 min

Visualizing the complex developer journey for JVM ecosystems

Bobur Umurzokov · LIVE

15:08 min

Audience questions on practical machine learning operational strategies

Lina Weichbrodt · LIVE

7:10 min

Exploring pathways into the machine learning engineering field

Jose Luis Latorre Millas · LIVE

More from WeAreDevelopers LIVE

WeAreDevelopers LIVE - Is CSS Catching Up to Flash? thumbnail
1:03:33
10 days ago

WeAreDevelopers LIVE - Is CSS Catching Up to Flash?

Chris Heilmann, Daniel Cranney, Bramus Van Damme

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along thumbnail
1:21:17
17 days ago

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along

Chris Heilmann, Daniel Cranney, Krzysztof Cieślak

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak thumbnail
4:23
19 days ago

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Open Source Agent Toolkits thumbnail
1:00:38
24 days ago

WeAreDevelopers LIVE - Open Source Agent Toolkits

Chris Heilmann, Daniel Cranney, Eddie Jaoude

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude thumbnail
6:16
26 days ago

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude

Chris Heilmann, Daniel Cranney

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail
5:46
about 1 month ago

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

Related videos

Alibaba Big Data and Machine Learning Technology thumbnail
43:57
LIVE • almost 6 years ago

Alibaba Big Data and Machine Learning Technology

Qiyang Duan

Overview of Machine Learning in Python thumbnail
57:46
LIVE • almost 3 years ago

Overview of Machine Learning in Python

Adrian Schmitt

Introduction to Azure Machine Learning thumbnail
54:27
LIVE • over 4 years ago

Introduction to Azure Machine Learning

Jose Luis Latorre Millas

Data Science in Retail thumbnail
58:21
LIVE • over 3 years ago

Data Science in Retail

Julian Joseph

Let's Get Aggregated: Custom UDAFs in Spark thumbnail
19:58
World Congress 2025 • about 1 year ago

Let's Get Aggregated: Custom UDAFs in Spark

Nikolai Nikolaev

Data Fabric in Action - How to enhance a Stock Trading App with ML and Data Virtualization thumbnail
45:58
LIVE • about 5 years ago

Data Fabric in Action - How to enhance a Stock Trading App with ML and Data Virtualization

Andreas Christian

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.