WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Oct 12, 2020

Alibaba Big Data and Machine Learning Technology

Qiyang Duan

How do you process half a million transactions per second without performance degradation? Discover the bare-metal Kubernetes and machine learning architecture powering Alibaba's legendary Double 11 festival.

Pause
Mute Enter Fullscreen
#1 about 9 min

Scaling big data infrastructure for extreme transaction volumes

Handling unprecedented global e-commerce demands requires deploying highly scalable containerized architectures.

#2 about 5 min

Exploring the big data and machine learning portfolio

Managing the complete data lifecycle is achieved through an integrated ecosystem bridging storage, processing, and visualization.

#3 about 5 min

Comparing distributed processing architectures for batch and streaming

Supporting immense analytic throughput and real-time computation involves specialized engines designed for variable workload types.

#4 about 6 min

Centralizing data engineering and operations with unified IDEs

Fragmented data engineering workflows are resolved by leveraging a centralized interface for governance, scripting, and pipeline scheduling.

#5 about 4 min

Architecting machine learning projects with the PAI platform

Accelerating algorithmic deployment requires offering tools that span from visual graph builders to code-first notebook environments.

#6 about 4 min

Leveraging templates and managed notebooks for data science

Removing infrastructure overhead allows teams to rely on managed instances and prebuilt model templates for rapid experimentation.

#7 about 2 min

Engaging developers through global artificial intelligence algorithm competitions

Encouraging algorithmic innovation involves creating structured community datasets and competitions for machine learning engineers globally.

#8 about 6 min

Executing queries and scheduling pipeline jobs within DataWorks

Centralizing data operations makes writing custom integrations and automating analytical transformations significantly more efficient.

#9 about 3 min

Constructing machine learning pipelines visually via PAI Studio

Bypassing complex coding for standard statistical models is possible by linking prebuilt algorithms in a visual canvas.

#10 about 5 min

Training neural networks and switching hardware inside notebooks

Optimizing compute costs during deep learning research is simplified by dynamically toggling between CPU and GPU hardware availability.

Matching moments

1:43 min

AWS infrastructure stack and data flow pipeline overview

Artem Volk Artem Volk +1 · WWC 2024

3:38 min

Architecting a unified data and machine learning workbench

Kapil Gupta Kapil Gupta · WWC 2025

4:49 min

Structuring compute and data services for AI models

Radu Vunvulea Radu Vunvulea · WWC 2025

3:03 min

Career evolution in data engineering and AI platforms

Maria Apazoglou · Coffee With Developers

2:03 min

Solving complex engineering challenges in artificial intelligence deployment

Nico Axtmann · WWC 2022

2:56 min

Introduction of panelists and their roles in AI engineering

Damandeep Kochhar Damandeep Kochhar +4 · WWC Europe 2026

More from WeAreDevelopers LIVE

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
5 days ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Node and Package Security thumbnail
1:08:30
5 days ago

WeAreDevelopers LIVE - Node and Package Security

Chris Heilmann, Daniel Cranney, Zbyszek Tenerowicz

WeAreDevelopers LIVE - Streaming HTML thumbnail
1:10:26
12 days ago

WeAreDevelopers LIVE - Streaming HTML

Chris Heilmann, Daniel Cranney, Jad Joubran

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe thumbnail
1:02:34
19 days ago

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe

Chris Heilmann, Daniel Cranney, Justin Halsall, Werner Jeschko, Nicole Jeschko, Mirco Tomić

WeAreDevelopers LIVE - Web Performance in an AI World thumbnail
1:07:59
26 days ago

WeAreDevelopers LIVE - Web Performance in an AI World

Chris Heilmann, Daniel Cranney

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko thumbnail
5:55
about 2 months ago

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

World Congress 2026 North America

AI Agents are Only as Smart as their Context: Building a Real-Time Context Engine at Intuit

Bharat Patel

Lead Software Engineer at Intuit

Bharat Patel
Open session

World Congress 2026 North America

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan
Open session

World Congress 2026 North America

Making Science Larger, not just Faster

Yuval Dvir

Commercial Executive, SandboxAQ

Yuval Dvir
Open session

World Congress 2026 North America

Look What Java Can Do Now: Live-Coding a GenAI MCP Server with the JAQ Stack

Suren Konathala

Senior Engineering Leader, Marketing & AdTech, AI GTM & Digital Experience Platforms, Java x AI

Suren Konathala
Open session

World Congress 2026 North America

Ship 10x Faster: AI-Powered Development with Claude Code and MCP Tools

Viktoria Semaan

Principal Technical Evangelist at Databricks

Viktoria Semaan
Open session

World Congress 2026 North America

Beyond the Code: Human-AI Synergies in Product Development

Ajita Kanchivakam Ananth

Staff Technical Program Manager at Google

Ajita Kanchivakam Ananth

Related videos

Data Fabric in Action - How to enhance a Stock Trading App with ML and Data Virtualization thumbnail
45:58
LIVE • almost 5 years ago

Data Fabric in Action - How to enhance a Stock Trading App with ML and Data Virtualization

Andreas Christian

Empowering Retail Through Applied Machine Learning thumbnail
22:41
WWC 2024 • almost 2 years ago

Empowering Retail Through Applied Machine Learning

Christoph Fassbach, Daniel Rohr

PySpark - Combining Machine Learning & Big Data thumbnail
43:26
LIVE • almost 6 years ago

PySpark - Combining Machine Learning & Big Data

Ayon Roy

Beyond GPT: Building Unified GenAI Platforms for the Enterprise of Tomorrow thumbnail
21:22
WWC 2025 • 12 months ago

Beyond GPT: Building Unified GenAI Platforms for the Enterprise of Tomorrow

Kapil Gupta

Industrializing your Data Science capabilities thumbnail
43:47
WWC 2021 • about 5 years ago

Industrializing your Data Science capabilities

Dubravko Dolic, Hüdaverdi Cakir

Developer Experience, Platform Engineering and AI powered Apps thumbnail
31:38
WWC 2024 • almost 2 years ago

Developer Experience, Platform Engineering and AI powered Apps

Ignacio Riesgo, Natale Vinto

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.