WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2023 • Oct 6, 2023

How We Built a Machine Learning-Based Recommendation System (And Survived to Tell the Tale)

Dora Petrella

A data science team replaced their underperforming SQL recommender with a Meta-Prod2Vec neural network. Learn how injecting article metadata solved long-tail recommendations and boosted product coverage by 45 percent.

Pause
Mute Enter Fullscreen
#1 about 2 min

Identifying the need for product replacement recommendations

Out-of-stock items drive customers to competitors unless relevant alternatives are immediately surfaceable.

#2 about 2 min

Understanding data sources in a complex b2b environment

Combining internal systems, website events, and purchase histories provides the foundational data for machine learning models.

#3 about 3 min

Defining multiple business use cases for recommendations

A flexible recommendation architecture serves diverse needs ranging from customer-facing web pages to internal order management tools.

#4 about 3 min

Evaluating limitations of legacy sql-based recommenders

Statistical formulas calculated continuously in sql struggle to provide accurate recommendations for newly listed or unpopular items.

#5 about 3 min

Adapting natural language algorithms for product recommendations

Applying prod2vec logic to user interaction sessions allows engineers to derive implicit training labels using sliding context windows.

#6 about 2 min

Training a neural network with one-hot encoded vectors

Feeding product combinations into a hidden layer model generates probability scores for items appearing within the same session.

#7 about 2 min

Improving long-tail predictions using meta-prod2vec algorithms

Concatenating category and brand metadata into product vectors significantly improves alternative matching for infrequently purchased items.

#8 about 1 min

Calculating vector distances to generate final alternatives

Measuring cosine similarity between geometric array representations allows the system to surface the most contextually relevant product replacements.

#9 about 2 min

Building the machine learning engineering pipeline

Connecting dbt transformations with vertex ai and orchestrating regular automated training runs ensures clean data flows into the elasticsearch storage.

#10 about 3 min

Evaluating model performance through offline metrics mapping

Testing against a production-like distribution utilizing ndcg and shadow traffic analysis confirms improvements in latency and catalog coverage.

#11 about 4 min

Visualizing product embeddings to validate clustering logic

Plotting multidimensional vectors in an open-source 3d projector tool confirms that related product attributes cluster accurately.

#12 about 2 min

Establishing pragmatic business baselines for model evaluation

Implementing a simple heuristic baseline metric sets a measurable quality gate and builds trust with non-technical stakeholders.

#13 about 2 min

Automating exploratory data analysis within training pipelines

Embedding automated html plotting directly into the model training pipeline prevents bad data from silently corrupting updated training runs.

#14 about 2 min

Team composition and realistic product launch timelines

Delivering an end-to-end recommendation system requires cross-functional collaboration and often exceeds initial baseline timeline estimates.

Matching moments

1:50 min

Real life recommendation systems and final project conclusions

Lutske van der Meer Lutske van der Meer · World Congress 2024

2:40 min

Understanding real-world recommendation systems in common platforms

Julian Joseph · LIVE

2:14 min

Assessing the future of AI in web performance optimization

Perf + AI

4:42 min

Predicting future retail experiences and avoiding technology hype

Alejandro Saucedo Alejandro Saucedo +2 · World Congress 2025

57 sec

Implementing predictive prefetch with machine learning

Jessica Janiuk · JS Congress

2:50 min

Driving subtle product enhancements with historical platform data

Alejandro Saucedo Alejandro Saucedo +2 · World Congress 2025

More from World Congress 2023

The year 3000, a brief history of Web Development thumbnail
25:30
over 2 years ago

The year 3000, a brief history of Web Development

Lorenzo Pieri

Demografie nutzen um jedem Bewerber das passende Angebot zu machen! thumbnail
8:52
almost 3 years ago

Demografie nutzen um jedem Bewerber das passende Angebot zu machen!

Nathalie Gierth

Guided Memory Management: Rust's Ownership Model thumbnail
0:18
almost 3 years ago

Guided Memory Management: Rust's Ownership Model

Stefan Baumgartner

Passwordless future: WebAuthn and Passkeys in practice thumbnail
32:32
almost 3 years ago

Passwordless future: WebAuthn and Passkeys in practice

Clemens Hübner

External Secrets Operator: the secrets management toolbox for self-sufficient teams thumbnail
30:07
almost 3 years ago

External Secrets Operator: the secrets management toolbox for self-sufficient teams

Moritz Johner

Building Reliable Serverless Applications with AWS CDK and Testing thumbnail
25:18
almost 3 years ago

Building Reliable Serverless Applications with AWS CDK and Testing

Raphael Manke

Related videos

Data Science in Retail thumbnail
58:21
LIVE • over 3 years ago

Data Science in Retail

Julian Joseph

What non-automotive Machine Learning projects can learn from automotive Machine Learning projects thumbnail
48:01
World Congress 2022 • over 4 years ago

What non-automotive Machine Learning projects can learn from automotive Machine Learning projects

Jan Zawadzki

Is my AI alive but brain-dead? How monitoring can tell you if your machine learning stack is still performing thumbnail
48:07
LIVE • almost 5 years ago

Is my AI alive but brain-dead? How monitoring can tell you if your machine learning stack is still performing

Lina Weichbrodt

Building Products in the era of GenAI thumbnail
59:39
LIVE • almost 3 years ago

Building Products in the era of GenAI

Julian Joseph

Deployed ML models need your feedback too thumbnail
37:24
World Congress 2021 • over 5 years ago

Deployed ML models need your feedback too

David Mosen

Anomaly Detection - Using unsupervised Machine Learning for detecting anomalies in customer base thumbnail
22:06
LIVE • over 6 years ago

Anomaly Detection - Using unsupervised Machine Learning for detecting anomalies in customer base

Lukas Kölbl

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.