WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Oct 12, 2020

Alibaba Big Data and Machine Learning Technology

Qiyang Duan

How do you process half a million transactions per second without performance degradation? Discover the bare-metal Kubernetes and machine learning architecture powering Alibaba's legendary Double 11 festival.

Pause
Mute Enter Fullscreen
#1 about 9 min

Scaling big data infrastructure for extreme transaction volumes

Handling unprecedented global e-commerce demands requires deploying highly scalable containerized architectures.

#2 about 5 min

Exploring the big data and machine learning portfolio

Managing the complete data lifecycle is achieved through an integrated ecosystem bridging storage, processing, and visualization.

#3 about 5 min

Comparing distributed processing architectures for batch and streaming

Supporting immense analytic throughput and real-time computation involves specialized engines designed for variable workload types.

#4 about 6 min

Centralizing data engineering and operations with unified IDEs

Fragmented data engineering workflows are resolved by leveraging a centralized interface for governance, scripting, and pipeline scheduling.

#5 about 4 min

Architecting machine learning projects with the PAI platform

Accelerating algorithmic deployment requires offering tools that span from visual graph builders to code-first notebook environments.

#6 about 4 min

Leveraging templates and managed notebooks for data science

Removing infrastructure overhead allows teams to rely on managed instances and prebuilt model templates for rapid experimentation.

#7 about 2 min

Engaging developers through global artificial intelligence algorithm competitions

Encouraging algorithmic innovation involves creating structured community datasets and competitions for machine learning engineers globally.

#8 about 6 min

Executing queries and scheduling pipeline jobs within DataWorks

Centralizing data operations makes writing custom integrations and automating analytical transformations significantly more efficient.

#9 about 3 min

Constructing machine learning pipelines visually via PAI Studio

Bypassing complex coding for standard statistical models is possible by linking prebuilt algorithms in a visual canvas.

#10 about 5 min

Training neural networks and switching hardware inside notebooks

Optimizing compute costs during deep learning research is simplified by dynamically toggling between CPU and GPU hardware availability.

Matching moments

1:43 min

AWS infrastructure stack and data flow pipeline overview

Artem Volk Artem Volk +1 · World Congress 2024

3:38 min

Architecting a unified data and machine learning workbench

Kapil Gupta Kapil Gupta · World Congress 2025

4:49 min

Structuring compute and data services for AI models

Radu Vunvulea Radu Vunvulea · World Congress 2025

3:03 min

Career evolution in data engineering and AI platforms

Maria Apazoglou · Coffee With Developers

1:29 min

Tech infrastructure capacity and AI product innovations

2:03 min

Solving complex engineering challenges in artificial intelligence deployment

Nico Axtmann · World Congress 2022

More from WeAreDevelopers LIVE

WeAreDevelopers LIVE - Is CSS Catching Up to Flash? thumbnail
1:03:33
12 days ago

WeAreDevelopers LIVE - Is CSS Catching Up to Flash?

Chris Heilmann, Daniel Cranney, Bramus Van Damme

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along thumbnail
1:21:17
19 days ago

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along

Chris Heilmann, Daniel Cranney, Krzysztof Cieślak

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak thumbnail
4:23
21 days ago

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Open Source Agent Toolkits thumbnail
1:00:38
26 days ago

WeAreDevelopers LIVE - Open Source Agent Toolkits

Chris Heilmann, Daniel Cranney, Eddie Jaoude

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude thumbnail
6:16
28 days ago

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude

Chris Heilmann, Daniel Cranney

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail
5:46
about 1 month ago

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

Related videos

Data Fabric in Action - How to enhance a Stock Trading App with ML and Data Virtualization thumbnail
45:58
LIVE • about 5 years ago

Data Fabric in Action - How to enhance a Stock Trading App with ML and Data Virtualization

Andreas Christian

Empowering Retail Through Applied Machine Learning thumbnail
22:41
World Congress 2024 • about 2 years ago

Empowering Retail Through Applied Machine Learning

Christoph Fassbach, Daniel Rohr

PySpark - Combining Machine Learning & Big Data thumbnail
43:26
LIVE • almost 6 years ago

PySpark - Combining Machine Learning & Big Data

Ayon Roy

Beyond GPT: Building Unified GenAI Platforms for the Enterprise of Tomorrow thumbnail
21:22
World Congress 2025 • about 1 year ago

Beyond GPT: Building Unified GenAI Platforms for the Enterprise of Tomorrow

Kapil Gupta

Industrializing your Data Science capabilities thumbnail
43:47
World Congress 2021 • over 5 years ago

Industrializing your Data Science capabilities

Dubravko Dolic, Hüdaverdi Cakir

Developer Experience, Platform Engineering and AI powered Apps thumbnail
31:38
World Congress 2024 • about 2 years ago

Developer Experience, Platform Engineering and AI powered Apps

Ignacio Riesgo, Natale Vinto

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.