WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Oct 12, 2020

Alibaba Big Data and Machine Learning Technology

Qiyang Duan

How do you process half a million transactions per second without performance degradation? Discover the bare-metal Kubernetes and machine learning architecture powering Alibaba's legendary Double 11 festival.

Pause
Mute Enter Fullscreen
#1 about 9 min

Scaling big data infrastructure for extreme transaction volumes

Handling unprecedented global e-commerce demands requires deploying highly scalable containerized architectures.

#2 about 5 min

Exploring the big data and machine learning portfolio

Managing the complete data lifecycle is achieved through an integrated ecosystem bridging storage, processing, and visualization.

#3 about 5 min

Comparing distributed processing architectures for batch and streaming

Supporting immense analytic throughput and real-time computation involves specialized engines designed for variable workload types.

#4 about 6 min

Centralizing data engineering and operations with unified IDEs

Fragmented data engineering workflows are resolved by leveraging a centralized interface for governance, scripting, and pipeline scheduling.

#5 about 4 min

Architecting machine learning projects with the PAI platform

Accelerating algorithmic deployment requires offering tools that span from visual graph builders to code-first notebook environments.

#6 about 4 min

Leveraging templates and managed notebooks for data science

Removing infrastructure overhead allows teams to rely on managed instances and prebuilt model templates for rapid experimentation.

#7 about 2 min

Engaging developers through global artificial intelligence algorithm competitions

Encouraging algorithmic innovation involves creating structured community datasets and competitions for machine learning engineers globally.

#8 about 6 min

Executing queries and scheduling pipeline jobs within DataWorks

Centralizing data operations makes writing custom integrations and automating analytical transformations significantly more efficient.

#9 about 3 min

Constructing machine learning pipelines visually via PAI Studio

Bypassing complex coding for standard statistical models is possible by linking prebuilt algorithms in a visual canvas.

#10 about 5 min

Training neural networks and switching hardware inside notebooks

Optimizing compute costs during deep learning research is simplified by dynamically toggling between CPU and GPU hardware availability.

Matching moments

1:43 min

AWS infrastructure stack and data flow pipeline overview

Artem Volk Artem Volk +1 · World Congress 2024

3:38 min

Architecting a unified data and machine learning workbench

Kapil Gupta Kapil Gupta · World Congress 2025

4:49 min

Structuring compute and data services for AI models

Radu Vunvulea Radu Vunvulea · World Congress 2025

3:03 min

Career evolution in data engineering and AI platforms

Maria Apazoglou · Coffee With Developers

1:29 min

Tech infrastructure capacity and AI product innovations

2:03 min

Solving complex engineering challenges in artificial intelligence deployment

Nico Axtmann · World Congress 2022

More from WeAreDevelopers LIVE

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude thumbnail
6:16
7 days ago

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude

Chris Heilmann, Daniel Cranney

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail
5:46
12 days ago

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

WeAreDevelopers - Of Agents and Robots thumbnail
1:04:42
13 days ago

WeAreDevelopers - Of Agents and Robots

Chris Heilmann, Daniel Cranney, Marc Plogas

WeAreDevelopers LIVE - How DevRel Makes Tech More Human thumbnail
48:06
20 days ago

WeAreDevelopers LIVE - How DevRel Makes Tech More Human

Chris Heilmann, Daniel Cranney

Fake or News: Smart Glasses in Court, Medical Advice Watches & Mushroom Electonics - Jason Torres thumbnail
7:41
21 days ago

Fake or News: Smart Glasses in Court, Medical Advice Watches & Mushroom Electonics - Jason Torres

Chris Heilmann, Daniel Cranney

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
about 1 month ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 24, 2026 · 11:00–11:30

Stage 1

Application-Defined Compute: Rethinking Infrastructure for AI Applications

Anurag Goel

Founder and CEO of Render

Anurag Goel
Open session

World Congress 2026 North America

September 25, 2026 · 09:00–09:30

Stage 7

AI Agents are Only as Smart as their Context: Building a Real-Time Context Engine at Intuit

Bharat Patel

Lead Software Engineer at Intuit

Bharat Patel
Open session

World Congress 2026 North America

September 25, 2026 · 09:00–09:30

Mainstage

Building AI that fits your business

Benny Chen

Co-Founder of Fireworks

Benny Chen
Open session

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 7

Designing High-Performance AI APIs: Lessons from Serving Millions of Real-Time Requests

Wayne Liu

Chief Growth Officer and Americas President of Perfect Corp.

Wayne Liu
Open session

World Congress 2026 North America

September 25, 2026 · 11:40–12:10

Stage 9

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan
Open session

World Congress 2026 North America

September 24, 2026 · 11:00–11:30

Stage 3

Making Science Larger, not just Faster

Yuval Dvir

Commercial Executive, SandboxAQ

Yuval Dvir

Related videos

Data Fabric in Action - How to enhance a Stock Trading App with ML and Data Virtualization thumbnail
45:58
LIVE • almost 5 years ago

Data Fabric in Action - How to enhance a Stock Trading App with ML and Data Virtualization

Andreas Christian

Empowering Retail Through Applied Machine Learning thumbnail
22:41
World Congress 2024 • about 2 years ago

Empowering Retail Through Applied Machine Learning

Christoph Fassbach, Daniel Rohr

PySpark - Combining Machine Learning & Big Data thumbnail
43:26
LIVE • almost 6 years ago

PySpark - Combining Machine Learning & Big Data

Ayon Roy

Beyond GPT: Building Unified GenAI Platforms for the Enterprise of Tomorrow thumbnail
21:22
World Congress 2025 • about 1 year ago

Beyond GPT: Building Unified GenAI Platforms for the Enterprise of Tomorrow

Kapil Gupta

Industrializing your Data Science capabilities thumbnail
43:47
World Congress 2021 • about 5 years ago

Industrializing your Data Science capabilities

Dubravko Dolic, Hüdaverdi Cakir

Developer Experience, Platform Engineering and AI powered Apps thumbnail
31:38
World Congress 2024 • about 2 years ago

Developer Experience, Platform Engineering and AI powered Apps

Ignacio Riesgo, Natale Vinto

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.