> Markdown version of [/videos/37-alibaba-big-data-and-machine-learning-technology?t=500](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology?t=500). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Alibaba Big Data and Machine Learning Technology

How do you process half a million transactions per second without performance degradation? Discover the bare-metal Kubernetes and machine learning architecture powering Alibaba's legendary Double 11 festival.

- **Speakers:** Qiyang Duan
- **Event:** WeAreDevelopers LIVE
- **Published:** October 12, 2020
- **Duration:** 43:57
- **URL:** https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology

## Summary

Alibaba's Big Data and artificial intelligence technology stack was incubated to solve extreme internal business demands, most notably managing the monumental scale of the Double 11 e-commerce festival. Handling up to half a million transactions per second and processing hundreds of petabytes of data daily requires unprecedented infrastructure agility. By migrating completely to a container-based, bare-metal Kubernetes architecture and leveraging stream processing engines like Apache Flink, the organization efficiently orchestrates massive real-time processing and complex machine learning workflows without performance degradation.

The core processing ecosystem surrounds MaxCompute, a flagship managed storage engine that scales linearly to exabyte levels while outperforming traditional distributed setups. However, the critical layer unifying this environment is DataWorks. Functioning as a comprehensive integrated development environment, DataWorks enables a true "DevOps for data" methodology. Instead of fracturing workflows across disconnected database consoles and command lines, engineers can ingest data, query efficiently, schedule task dependencies, and govern data lineage from a single unified command center.

Bridging these foundational data pipelines to predictive execution, the Platform for AI (PAI) abstracts the complexities of deploying production machine learning models. The framework accommodates varying engineering cadences by integrating tools like PAI Studio for visual, drag-and-drop model orchestration alongside the Data Science Workshop (DSW) for code-heavy generation. DSW specifically empowers developers with a managed Jupyter Lab environment offering seamless, on-the-fly computational switching between CPU and GPU hardware, entirely eliminating administrative overhead during intense model iteration. Ultimately, these advanced predictive models are rendered as scalable REST API endpoints through the Elastic Algorithm Service, completing a seamless pipeline from raw e-commerce logs to real-world operational intelligence.

**Keywords:** alibaba cloud architecture, apache flink stream processing, maxcompute data warehousing, dataworks unified ide, big data devops, data governance mapping, visual drag-and-drop modeling, jupyter notebook environments, cpu to gpu resource switching, elastic algorithm service, e-commerce real-time analytics, kubernetes container orchestration, hadoop cluster migration, machine learning pipeline orchestration, tensorflow model training

## Chapters

1. **Scaling big data infrastructure for extreme transaction volumes** (00:17) — Handling unprecedented global e-commerce demands requires deploying highly scalable containerized architectures.
1. **Exploring the big data and machine learning portfolio** (08:20) — Managing the complete data lifecycle is achieved through an integrated ecosystem bridging storage, processing, and visualization.
1. **Comparing distributed processing architectures for batch and streaming** (12:38) — Supporting immense analytic throughput and real-time computation involves specialized engines designed for variable workload types.
1. **Centralizing data engineering and operations with unified IDEs** (16:40) — Fragmented data engineering workflows are resolved by leveraging a centralized interface for governance, scripting, and pipeline scheduling.
1. **Architecting machine learning projects with the PAI platform** (21:46) — Accelerating algorithmic deployment requires offering tools that span from visual graph builders to code-first notebook environments.
1. **Leveraging templates and managed notebooks for data science** (25:40) — Removing infrastructure overhead allows teams to rely on managed instances and prebuilt model templates for rapid experimentation.
1. **Engaging developers through global artificial intelligence algorithm competitions** (29:00) — Encouraging algorithmic innovation involves creating structured community datasets and competitions for machine learning engineers globally.
1. **Executing queries and scheduling pipeline jobs within DataWorks** (31:01) — Centralizing data operations makes writing custom integrations and automating analytical transformations significantly more efficient.
1. **Constructing machine learning pipelines visually via PAI Studio** (36:20) — Bypassing complex coding for standard statistical models is possible by linking prebuilt algorithms in a visual canvas.
1. **Training neural networks and switching hardware inside notebooks** (38:58) — Optimizing compute costs during deep learning research is simplified by dynamically toggling between CPU and GPU hardware availability.

## Related Moments

- [AWS infrastructure stack and data flow pipeline overview](https://www.wearedevelopers.com/videos/944-building-the-platform-for-providing-ml-predictions-based-on-real-time-player-activity) (from "Building the platform for providing ML predictions based on real-time player activity")
- [Architecting a unified data and machine learning workbench](https://www.wearedevelopers.com/videos/1525-beyond-gpt-building-unified-genai-platforms-for-the-enterprise-of-tomorrow) (from "Beyond GPT: Building Unified GenAI Platforms for the Enterprise of Tomorrow")
- [Structuring compute and data services for AI models](https://www.wearedevelopers.com/videos/1613-reference-architecture-of-ai-in-the-cloud) (from "Reference Architecture of AI in the Cloud")
- [Career evolution in data engineering and AI platforms](https://www.wearedevelopers.com/videos/1209-coffee-with-developers-maria-apazoglou) (from "Coffee with Developers - Maria Apazoglou")
- [Solving complex engineering challenges in artificial intelligence deployment](https://www.wearedevelopers.com/videos/392-mlops-what-s-the-deal-behind-it) (from "MLOps - What’s the deal behind it?")
- [Introduction of panelists and their roles in AI engineering](https://www.wearedevelopers.com/videos/100310-data-the-deciding-factor-in-ai-success) (from "Data: The Deciding Factor in AI Success")

## Related Articles

- [Stephan Gillich - Bringing AI Everywhere](https://www.wearedevelopers.com/magazine/489-stephan-gillich-bringing-ai-everywhere)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)

## Related Jobs

- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [Lead Software Engineer - Data Engineering](https://www.wearedevelopers.com/jobs/ext/2000968-lead-software-engineer-data-engineering) at **Dynatrace**
- [Principal Software Engineer, Enterprise AI Platform](https://www.wearedevelopers.com/jobs/ext/1467292-principal-software-engineer-enterprise-ai-platform) at **GitHub**
- [Staff, Machine Learning Engineer (L4)](https://www.wearedevelopers.com/jobs/ext/1202639-staff-machine-learning-engineer-l4) at **Twilio**
- [Artificial Intelligence (AI)](https://www.wearedevelopers.com/jobs/ext/1952055-artificial-intelligence-ai) at **Twilio**
- [Principal Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1706410-principal-machine-learning-engineer) at **Almedia**