WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Oct 13, 2021

Is my AI alive but brain-dead? How monitoring can tell you if your machine learning stack is still performing

Lina Weichbrodt

Is your machine learning model technically functional but quietly damaging your business? Discover how to spot a brain-dead AI using your existing observability stack to prevent silent degradation.

Pause
Mute Enter Fullscreen
#1 about 2 min

Defining success and business requirements in machine learning

Since vague business requirements often derail machine learning projects, developers must establish precise performance criteria upfront.

#2 about 2 min

Example business case for predicting loan application rejections

Generating internal rejection predictions saves businesses from incurring high monthly API processing costs via external agencies.

#3 about 5 min

Choosing precision and recall metrics for loan optimization

To balance financial savings with accuracy, teams adjust prediction thresholds across standard precision and recall metrics.

#4 about 2 min

Applying gradient boosted trees for tabular data training

Since tabular database columns defeat many deep learning approaches, ensemble algorithms provide vastly superior training speed.

#5 about 3 min

Selecting monitoring infrastructure stacks for machine learning operations

Instead of adopting complex proprietary platforms, introductory machine learning teams can repurpose existing engineering dashboard tools.

#6 about 6 min

Monitoring correct model outcomes in delayed production environments

When automated decisions obscure natural outcomes, deploying an untouched traffic slice provides necessary baseline performance metrics.

#7 about 3 min

Translating stakeholder concerns into actionable production monitoring signals

To bridge the gap with non-technical partners, developers translate explicit worst-case fears into monitored production alerts.

#8 about 4 min

Tracking model response distributions when true validations fail

Because ground truth labels arrive late, tracking statistical distance across output distributions reveals silent inference failures.

#9 about 3 min

Implementing common sense quality heuristics for model outputs

When deep statistical validation is impossible, human-understandable proxy metrics immediately highlight degraded algorithmic user experiences.

#10 about 2 min

Identifying pipeline discrepancies through live input feature monitoring

To catch hidden transformations, comparing offline training batches against live API traffic prevents silent feature engineering regressions.

#11 about 4 min

Differentiating model quality metrics from incident detection indicators

By prioritizing endpoint impacts over mathematical accuracy, teams surface critical operational failures before experiencing complete system degradation.

#12 about 16 min

Audience questions on practical machine learning operational strategies

Addressing real-world constraints clarifies delayed feedback management, ideal modeling techniques, and optimal pathways into data engineering.

Matching moments

4:17 min

Overcoming limitations in current machine learning monitoring tools

David Mosen · WWC 2021

4:01 min

Transitioning artificial intelligence into operational business environments

Stefan Donsa Stefan Donsa +1 · LIVE

1:05 min

Implementing monitoring and observability for AI software deployments

Alejandro Saucedo Alejandro Saucedo · WWC 2025

2:24 min

Monitoring AI model quality and execution performance

Jemiah Sius Jemiah Sius +1 · WWC Europe 2026

3:08 min

Maintaining and monitoring machine learning models in production

Natalie Pistunovich · LIVE

4:20 min

Designing architectures for automated machine learning lifecycles

David Mosen · WWC 2021

More from WeAreDevelopers LIVE

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
5 days ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Node and Package Security thumbnail
1:08:30
5 days ago

WeAreDevelopers LIVE - Node and Package Security

Chris Heilmann, Daniel Cranney, Zbyszek Tenerowicz

WeAreDevelopers LIVE - Streaming HTML thumbnail
1:10:26
12 days ago

WeAreDevelopers LIVE - Streaming HTML

Chris Heilmann, Daniel Cranney, Jad Joubran

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe thumbnail
1:02:34
19 days ago

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe

Chris Heilmann, Daniel Cranney, Justin Halsall, Werner Jeschko, Nicole Jeschko, Mirco Tomić

WeAreDevelopers LIVE - Web Performance in an AI World thumbnail
1:07:59
26 days ago

WeAreDevelopers LIVE - Web Performance in an AI World

Chris Heilmann, Daniel Cranney

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko thumbnail
5:55
about 2 months ago

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

World Congress 2026 North America

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan
Open session

World Congress 2026 North America

No Single Model to Rule Them All: Building Resilient AI Agents Across Open & Closed LLMs

Emmanuel Acheampong

Senior Manager Developer Relations at Crusoe AI

Emmanuel Acheampong
Open session

World Congress 2026 North America

Reinventing Testing Practices in the AI Era

Eric Deandrea

Java Champion & Senior Principal Software Engineer, IBM

Eric Deandrea
Open session

World Congress 2026 North America

AI Decision Observability: Enabling Transparency and Trust in Intelligent Systems

Amjad Shaikh, Soumil Mandal

Amjad Shaikh
Soumil Mandal
Open session

World Congress 2026 North America

Proactive AI That Doesn’t Annoy Users: Building Context-Aware Notification Systems

Raju Dandigam Dandigam

Engineering Manager at Navan

Raju Dandigam Dandigam
Open session

World Congress 2026 North America

AI ROI: The Hard Unit Economics of AI-Native Engineering

Manu Gurudatha

Manu Gurudatha, VP of Engineering at PagerDuty

Manu Gurudatha

Related videos

Deployed ML models need your feedback too thumbnail
37:24
WWC 2021 • about 5 years ago

Deployed ML models need your feedback too

David Mosen

Why LLMs Need Observability and How to Do It thumbnail
32:41
Europe 2026 Virtual • about 1 month ago

Why LLMs Need Observability and How to Do It

Rahul Gupta

DevOps for Machine Learning thumbnail
44:40
WWC 2021 • about 5 years ago

DevOps for Machine Learning

Hauke Brammer

How AI Models Get Smarter thumbnail
35:16
WWC 2025 • about 1 year ago

How AI Models Get Smarter

Ankit Patel

Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source thumbnail
29:50
WWC Europe 2026 • about 1 month ago

Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source

Viktoria Semaan

Unlocking the AI Black Box: Building Trust in the Era of Agentic Production thumbnail
27:40
WWC Europe 2026 • about 1 month ago

Unlocking the AI Black Box: Building Trust in the Era of Agentic Production

Jemiah Sius, Harry Kimpel

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.