> Markdown version of [/videos/944-building-the-platform-for-providing-ml-predictions-based-on-real-time-player-activity?t=337](https://www.wearedevelopers.com/videos/944-building-the-platform-for-providing-ml-predictions-based-on-real-time-player-activity?t=337). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Building the platform for providing ML predictions based on real-time player activity

How do you process millions of telemetry events to personalize gameplay in under 500 milliseconds? Explore Phoenix Games' decoupled, fault-tolerant machine learning architecture on AWS.

- **Speakers:** [Artem Volk](https://www.wearedevelopers.com/@artem-volk), [Fabian Zillgens](https://www.wearedevelopers.com/@fabian-zillgens)
- **Event:** World Congress 2024
- **Published:** August 20, 2024
- **Duration:** 30:57
- **URL:** https://www.wearedevelopers.com/videos/944-building-the-platform-for-providing-ml-predictions-based-on-real-time-player-activity

## Summary

Phoenix Games engineered a lightweight event-streaming architecture on AWS to personalize gameplay and in-game shop offers in real time. Designed to be unobtrusive for game studios, the platform avoids heavy client SDKs by utilizing a pull-based API where games request customizations on the fly. Incoming telemetry events—up to 1.5 million per hour per title—are ingested via Amazon Kinesis, serving as a critical decoupling layer that prevents localized failures from halted data pipelines.

The system relies on a dual-processing approach to balance speed and historical context. Offline batching is managed by Apache Spark on EMR for long-term performance analytics and lifetime spend tracking, while Apache Flink SQL drives the online processing layer. Flink effortlessly updates real-time user profiles with fresh telemetry within roughly 500 milliseconds. When a player triggers a relevant event, the game queries a decision-making engine powered by AWS Lambda, allowing machine learning models to seamlessly return tailored content, adjust difficulty, or recommend targeted visual bundles.

By fully decoupling telemetry ingestion from model deployment, the architecture empowers data scientists to autonomously launch, monitor, and rollback their own models using standardized Terraform modules. Built-in systemic fault tolerance ensures that if a model misbehaves or latency delays the payload, the game safely discards the customization without crashing. Moreover, utilizing rigorous Redshift data partitioning, payload compression, and telemetry batching keeps the overall AWS infrastructure highly scalable and cost-efficient.

**Keywords:** real-time machine learning, player telemetry ingestion, apache flink streaming, apache spark batch processing, amazon kinesis streams, aws lambda inference, dynamic game customization, telemetry batching, data pipeline decoupling, terraform model automation, live ops personalization, cloud cost optimization, pull-based delivery api

## Chapters

1. **Overview of Phoenix Games and real-time customization goals** (01:29) — The challenge of adapting player experiences in real-time requires fast data iterations and automated process pipelines.
1. **Customizing player experiences in the Emergency HQ title** (02:39) — Machine learning models apply dynamic gameplay modifications to surface relevant store purchases alongside tailored mission difficulties.
1. **Microservice architecture requirements for unobtrusive event collection** (04:03) — Delivering dynamic game features demands simple cross-studio integration without heavy client SDK dependencies.
1. **AWS infrastructure stack and data flow pipeline overview** (05:37) — A simplified architecture highlights the event collection, offline data processing, and machine learning customization mechanisms.
1. **Managing raw game event execution and payload structures** (07:21) — Handling millions of out-of-order game stream instances requires distinct tracking of real event occurrence timestamps versus upload times.
1. **Comparing offline data analytics with online stream processing** (09:26) — Utilizing Apache Spark extracts long-term insights while real-time user activity is handled via unbound Apache Flink queries.
1. **Managing fast real-time player states using profile APIs** (11:31) — Storing live active user snapshot profiles enables rapid integration of both historical offline data and instant inferences.
1. **Demonstrating pipeline latency thresholds during player purchases** (12:33) — Tracking a native player transaction through Flink visualizes how the profile snapshot API updates within hundreds of milliseconds.
1. **Delivering dynamic game packages using external decision models** (14:15) — Decoupled AWS lambda functions empower independent machine learning calculations and simple rule-based feature testing configurations.
1. **Analyzing customization payloads alongside model inference decisions** (15:55) — Displaying specific package deals relies on observing distinct machine learning delays and debugging underlying player delay logic.
1. **Enabling autonomous deployments and independent monitoring controls** (17:30) — Empowering data scientists with Terraform dashboards minimizes blockages while securing the pipeline against flawed algorithm updates.
1. **Measuring API infrastructure limits and platform pipeline latency** (18:43) — Analyzing sub-second turnaround constraints uncovers operational bottlenecks when calling endpoints and executing live updates simultaneously.
1. **Evaluating deployment pipeline compromises and infrastructure scaling costs** (19:34) — Balancing the advantages of resilient decoupled microservices highlights fundamental trade-offs surrounding duplicate data and high storage costs.
1. **Processing extreme event volumes and validating machine operations** (24:04) — Operating at massive scale emphasizes the importance of data verification to prevent custom features from hurting core business revenue.
1. **Executing automated model generation and dynamic user packaging** (25:34) — Empowering data analysts to build localized cloud containers directly enables specific item delivery configurations tailored to individual gamers.
1. **Implementing Lambda request load balancing and player privacy controls** (27:55) — Handling high volume interactions requires strict AWS container load provisioning and limited personal asset storage tracking parameters.

## Related Moments

- [Shifting data analytics from monetization to player experience design](https://www.wearedevelopers.com/videos/176-how-data-is-shaping-our-games) (from "How Data is Shaping our Games")
- [Modernizing legacy applications for real-time streaming data consumption](https://www.wearedevelopers.com/videos/1367-unlocking-value-from-data-the-key-to-smarter-business-decisions) (from "Unlocking Value from Data: The Key to Smarter Business Decisions-")
- [Optimizing system scalability and real-time push capabilities](https://www.wearedevelopers.com/videos/494-the-rise-of-reactive-microservices) (from "The Rise of Reactive Microservices")
- [Integrating data analytics and machine learning as interactive elements](https://www.wearedevelopers.com/videos/176-how-data-is-shaping-our-games) (from "How Data is Shaping our Games")
- [Scaling experimental game concepts into production fitness products](https://www.wearedevelopers.com/videos/100140-playing-pong-on-a-shoulder-press-machine) (from "Playing Pong on a shoulder press machine")
- [Behavioral profiling and creating custom player types in games](https://www.wearedevelopers.com/videos/176-how-data-is-shaping-our-games) (from "How Data is Shaping our Games")

## Related Articles

- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering)
- [Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)](https://www.wearedevelopers.com/magazine/745-why-event-driven-architecture-isn-t-about-speed-and-when-you-actually-need-it)
- [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix)

## Related Jobs

- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [Principal Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1706410-principal-machine-learning-engineer) at **Almedia**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1377841-machine-learning-engineer) at **Almedia**
- [Senior Software Engineer, Data](https://www.wearedevelopers.com/jobs/48273-senior-software-engineer-data) at **Sportradar Media Services GmbH**
- [Lead Software Engineer - Data Engineering](https://www.wearedevelopers.com/jobs/ext/2000968-lead-software-engineer-data-engineering) at **Dynatrace**
- [Staff, Machine Learning Engineer (L4)](https://www.wearedevelopers.com/jobs/ext/1202639-staff-machine-learning-engineer-l4) at **Twilio**