> Markdown version of [/videos/752-how-we-built-a-machine-learning-based-recommendation-system-and-survived-to-tell-the-tale?t=882](https://www.wearedevelopers.com/videos/752-how-we-built-a-machine-learning-based-recommendation-system-and-survived-to-tell-the-tale?t=882). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# How We Built a Machine Learning-Based Recommendation System (And Survived to Tell the Tale)

A data science team replaced their underperforming SQL recommender with a Meta-Prod2Vec neural network. Learn how injecting article metadata solved long-tail recommendations and boosted product coverage by 45 percent.

- **Speakers:** Dora Petrella
- **Event:** World Congress 2023
- **Published:** October 6, 2023
- **Duration:** 27:23
- **URL:** https://www.wearedevelopers.com/videos/752-how-we-built-a-machine-learning-based-recommendation-system-and-survived-to-tell-the-tale

## Summary

When customers encounter out-of-stock items, providing highly relevant alternatives is critical to preventing lost sales. To solve this, a data science team replaced an underperforming SQL-based Jaccard similarity recommender with a flexible, machine learning-based recommendation engine. The system was designed to serve an e-commerce website, an app, an order management UI, and internal listing decisions. The core solution adapted the Word2Vec mathematical model into a Prod2Vec algorithm, generating implicit training labels by tracking products viewed together within individual user session context windows. By capturing nearby pairwise combinations and contrasting them with randomly sampled negative examples, the model learns the probability of specific items appearing sequentially.

To maximize predictive power, the team ultimately implemented Meta-Prod2Vec, which injects article metadata—such as category, subcategory, and brand—directly into the neural network. This methodological shift generated vastly superior recommendations for the "long tail" of rarely purchased or new articles. The resulting network outputs 128-dimensional vector representations, enabling rapid, mathematically precise alternative suggestions via cosine similarity. Managing this workflow required an automated infrastructure utilizing dbt for data transformation, TensorFlow for model training, and Vertex AI with Kubeflow for orchestration, while Elasticsearch handled the high-speed vector similarity sorting in production.

Several pragmatic insights emerged to guide future data science workflows. Establishing a simple "business baseline"—such as generating a randomized SQL recommendation from a specific subcategory—provides an intuitive quality gate that makes benchmarking clearer for non-technical stakeholders. Additionally, relying solely on upfront exploratory data analysis leaves pipelines vulnerable to silent failures. Integrating automated data distribution plots directly into the continuous training pipeline ensures persistent visibility into data health, preventing the model from training on corrupted or empty datasets. Finally, utilizing mirrored live traffic before conducting formal AB testing allows technical metrics, like latency and coverage, to be safely validated, ultimately proving that the new model increased product coverage by 45 percent.

**Keywords:** machine learning, recommendation system, prod2vec, word2vec, vertex ai, kubeflow, tensorflow, dbt, vector similarity, elasticsearch, long-tail articles, metadata integration, exploratory data analysis, business baseline, user session tracking, embedding projector, data pipeline automation

## Chapters

1. **Identifying the need for product replacement recommendations** (00:14) — Out-of-stock items drive customers to competitors unless relevant alternatives are immediately surfaceable.
1. **Understanding data sources in a complex b2b environment** (02:11) — Combining internal systems, website events, and purchase histories provides the foundational data for machine learning models.
1. **Defining multiple business use cases for recommendations** (03:12) — A flexible recommendation architecture serves diverse needs ranging from customer-facing web pages to internal order management tools.
1. **Evaluating limitations of legacy sql-based recommenders** (05:40) — Statistical formulas calculated continuously in sql struggle to provide accurate recommendations for newly listed or unpopular items.
1. **Adapting natural language algorithms for product recommendations** (08:05) — Applying prod2vec logic to user interaction sessions allows engineers to derive implicit training labels using sliding context windows.
1. **Training a neural network with one-hot encoded vectors** (10:55) — Feeding product combinations into a hidden layer model generates probability scores for items appearing within the same session.
1. **Improving long-tail predictions using meta-prod2vec algorithms** (12:21) — Concatenating category and brand metadata into product vectors significantly improves alternative matching for infrequently purchased items.
1. **Calculating vector distances to generate final alternatives** (13:54) — Measuring cosine similarity between geometric array representations allows the system to surface the most contextually relevant product replacements.
1. **Building the machine learning engineering pipeline** (14:42) — Connecting dbt transformations with vertex ai and orchestrating regular automated training runs ensures clean data flows into the elasticsearch storage.
1. **Evaluating model performance through offline metrics mapping** (16:31) — Testing against a production-like distribution utilizing ndcg and shadow traffic analysis confirms improvements in latency and catalog coverage.
1. **Visualizing product embeddings to validate clustering logic** (19:24) — Plotting multidimensional vectors in an open-source 3d projector tool confirms that related product attributes cluster accurately.
1. **Establishing pragmatic business baselines for model evaluation** (22:58) — Implementing a simple heuristic baseline metric sets a measurable quality gate and builds trust with non-technical stakeholders.
1. **Automating exploratory data analysis within training pipelines** (24:27) — Embedding automated html plotting directly into the model training pipeline prevents bad data from silently corrupting updated training runs.
1. **Team composition and realistic product launch timelines** (26:15) — Delivering an end-to-end recommendation system requires cross-functional collaboration and often exceeds initial baseline timeline estimates.

## Related Moments

- [Real life recommendation systems and final project conclusions](https://www.wearedevelopers.com/videos/1014-machine-learning-101-where-to-begin) (from "Machine learning 101: Where to begin?")
- [Understanding real-world recommendation systems in common platforms](https://www.wearedevelopers.com/videos/586-data-science-in-retail) (from "Data Science in Retail")
- [Assessing the future of AI in web performance optimization](https://www.wearedevelopers.com/videos/1771-ai-is-an-electric-bike-for-the-brain-stoyan-stefanov) (from "AI is an Electric Bike for the Brain - Stoyan Stefanov")
- [Predicting future retail experiences and avoiding technology hype](https://www.wearedevelopers.com/videos/1702-intelligence-everywhere-the-future-of-consumer-tech) (from "Intelligence Everywhere: The Future of Consumer Tech")
- [Implementing predictive prefetch with machine learning](https://www.wearedevelopers.com/videos/287-state-of-angular) (from "State of Angular")
- [Driving subtle product enhancements with historical platform data](https://www.wearedevelopers.com/videos/1702-intelligence-everywhere-the-future-of-consumer-tech) (from "Intelligence Everywhere: The Future of Consumer Tech")

## Related Articles

- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)
- [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)

## Related Jobs

- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [Principal Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1706410-principal-machine-learning-engineer) at **Almedia**
- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1355348-machine-learning-engineer) at **TWILIO**
- [Principal Software Engineer, Enterprise AI Platform](https://www.wearedevelopers.com/jobs/ext/1467292-principal-software-engineer-enterprise-ai-platform) at **GitHub**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1377841-machine-learning-engineer) at **Almedia**