> Markdown version of [/videos/161-deployed-ml-models-need-your-feedback-too](https://www.wearedevelopers.com/videos/161-deployed-ml-models-need-your-feedback-too). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Deployed ML models need your feedback too

Standard infrastructure tracking won't catch concept drift in your ML models. Learn how to build continuous feedback loops that directly link ground-truth performance to business OKRs.

- **Speakers:** David Mosen
- **Event:** World Congress 2021
- **Published:** June 28, 2021
- **Duration:** 37:24
- **URL:** https://www.wearedevelopers.com/videos/161-deployed-ml-models-need-your-feedback-too

## Summary

Transitioning to advanced MLOps maturity demands moving beyond standard continuous integration and delivery pipelines to embrace continuous training and targeted production monitoring. While infrastructure tracking is well-established, validating active models requires isolating complex system decay like input drift and concept drift. Concept drift creates unique challenges by fundamentally altering the relationship between input variables and output targets, meaning engineering teams must completely re-annotate historical relationships to retrain models rather than simply aggregating higher volumes of drift-impacted data.

Operating resilient production intelligence relies entirely on establishing robust feedback loops capable of capturing ground-truth performance directly from user interactions. Real-world application delays dictate the effectiveness of these loops; real-time validation in e-commerce provides immediate optimization value, whereas financial demand forecasting imposes inherent structural monitoring lag that teams must account for administratively. Standardized managed services such as Google Vertex AI and Seldon Core Analytics attempt to streamline model auditing by binding endpoints to specific Kubernetes architectures or data warehouses, yet heavily struggle to map isolated algorithmic metrics (such as classification errors) directly to higher-level business OKRs without rigid infrastructure lock-in.

To bridge the divide between localized technical execution and enterprise value, modern deployments require agnostic evaluation platforms integrated natively with execution APIs. Operating with adaptive YAML profiling tools to evaluate proxy metrics alongside live feedback payloads—such as securely matching prediction correlation IDs against monthly customer churn telemetry—helps dissolve legacy operational silos separating developers from business stakeholders. Pivoting from merely observing schema-less proxy inputs toward rigorously verifying universal analytical performance ultimately anchors specialized machine learning assets directly to corporate financial health and customer retention impact.

**Keywords:** mlops continuous training, concept drift mitigation, automated feedback loops, ground-truth model performance, business OKR correlation, schema-less data monitoring, google vertex api evaluation, seldon core analytics, aws personalize tracking, rolling evaluation windows, prediction correlation ids, model deployment strategies, machine learning telemetry, prediction service monitoring, model metric degradation

## Chapters

1. **Defining machine learning operations and continuous training pipelines** (00:03) — Extending continuous integration and delivery with continuous training practices keeps algorithms performing accurately in live production environments.
1. **Transitioning through maturity levels of machine learning operations** (02:03) — Reaching full deployment automation requires active system monitoring and intelligent feedback loops to trigger retraining cycles automatically.
1. **Designing architectures for automated machine learning lifecycles** (04:41) — Integrating feature stores and prediction services within code repositories highlights the necessity of robust monitoring for deployment stability.
1. **Addressing input distribution changes and concept drift** (09:02) — Shifting baselines in input variables or their initial relationship to target outputs mandate teams to isolate algorithmic drift and relabel dataset pipelines.
1. **Correlating technical performance with high-level business metrics** (12:29) — Directly monitoring proxy metrics and model degradation surfaces critical insights into overall system stability and key business performance indicators.
1. **Managing delays and automation in model feedback collection** (16:49) — Connecting automated feedback streams to decoupled evaluation routines ensures teams can detect and remediate concept drift before prediction staleness compounds.
1. **Overcoming limitations in current machine learning monitoring tools** (19:50) — Current managed cloud environments and open-source stacks frequently lack business-layer monitoring loops and require extensive engineering configurations to maintain.
1. **Integrating native APIs for custom model monitoring profiles** (24:07) — Abstracting assessment through flexible model profiles allows engineering teams to track complex regression metrics directly across arbitrary management platforms.
1. **Visualizing model metrics and business indicators in context** (27:55) — Combining real-time technical health statistics with delayed business trends like user churn establishes a holistic view of overall software accuracy.
1. **Standardizing machine learning deployments for varying schemas and environments** (32:09) — Shifting deployment models requires teams to leverage proxy metrics when adapting standardized codebase templates to edge computing hardware or unstructured environments.

## Related Moments

- [Transitioning artificial intelligence into operational business environments](https://www.wearedevelopers.com/videos/111-detecting-money-laundering-with-ai) (from "Detecting Money Laundering with AI")
- [Defining MLOps and its role in production systems](https://www.wearedevelopers.com/videos/825-mlops-on-kubernetes-exploring-argo-workflows) (from "MLOps on Kubernetes: Exploring Argo Workflows")
- [Audience questions on practical machine learning operational strategies](https://www.wearedevelopers.com/videos/262-is-my-ai-alive-but-brain-dead-how-monitoring-can-tell-you-if-your-machine-learning-stack-is-still-performing) (from "Is my AI alive but brain-dead? How monitoring can tell you if your machine learning stack is still performing")
- [Maintaining and monitoring machine learning models in production](https://www.wearedevelopers.com/videos/347-mlops-and-ai-driven-development) (from "MLOps and AI Driven Development")
- [Bridging the gap between model management and devops](https://www.wearedevelopers.com/videos/1152-ai-model-management-life-circles-ml-ops-for-generative-ai-models-from-research-to-deployment) (from "AI Model Management Life Circles: ML Ops For Generative AI Models From Research to Deployment")
- [Navigating the complexities of machine learning model lifecycles](https://www.wearedevelopers.com/videos/1657-dataforce-studio) (from "DataForce Studio")

## Related Articles

- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [Stephan Gillich - Bringing AI Everywhere](https://www.wearedevelopers.com/magazine/489-stephan-gillich-bringing-ai-everywhere)

## Related Jobs

- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/588393-machine-learning-engineer) at **Twilio**
- [Staff, Machine Learning Engineer (L4)](https://www.wearedevelopers.com/jobs/ext/1202639-staff-machine-learning-engineer-l4) at **Twilio**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1355348-machine-learning-engineer) at **TWILIO**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1597388-machine-learning-engineer) at **ZEISS Group**
- [Principal Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1706410-principal-machine-learning-engineer) at **Almedia**
- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**