> Markdown version of [/videos/1518-solving-the-puzzle-leveraging-machine-learning-for-effective-root-cause-analysis?t=929](https://www.wearedevelopers.com/videos/1518-solving-the-puzzle-leveraging-machine-learning-for-effective-root-cause-analysis?t=929). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Solving the puzzle: Leveraging machine learning for effective root cause analysis

Are traditional statistical methods failing your complex manufacturing data? Discover how pairing LightGBM with Explainable AI cuts root cause analysis from days to under two hours.

- **Speakers:** [Bernhard](https://www.wearedevelopers.com/@bernhard), [Varsha Venugopal](https://www.wearedevelopers.com/@varsha-venugopal)
- **Event:** World Congress 2025
- **Published:** August 20, 2025
- **Duration:** 24:31
- **URL:** https://www.wearedevelopers.com/videos/1518-solving-the-puzzle-leveraging-machine-learning-for-effective-root-cause-analysis

## Summary

Modern manufacturing, such as the production of high-NA EUV lithography optics, involves immensely complex systems that generate massive, high-dimensional datasets. When production defects arise, traditional statistical methods like linear regression or ANOVA often struggle, as they assume normally distributed parameters and frequently drop significant portions of data due to high imbalance or missing values. To effectively isolate faults within these complex systems, engineering teams require advanced strategies capable of organically handling noisy, non-linear, and real-world variables.

By leveraging Machine Learning combined with Explainable AI (XAI), teams can transform opaque algorithms into transparent discovery engines. Utilizing the LightGBM framework allows practitioners to natively process structured, tabular data with missing values and discrete categoricals without heavy preprocessing. Once the model is trained, applying SHapley Additive exPlanations (SHAP) reveals both broad global feature significance and highly granular local insights. This structure accurately pinpoints the exact threshold where specific process variables—such as duration or temperature—exponentially increase the probability of a defect, highlighting anomalies that conventional domain knowledge might easily overlook.

Rather than treating AI as an automated replacement, packaging these methodologies into an internal, self-service process investigation tool—built combining Python, Shiny, and Plotly—strongly empowers domain experts. Human-centric ML design combined with transparent communication regarding a model's limitations (such as differentiating correlation from causation) is vital for driving engineering trust and adoption. Simplifying the interface to focus strictly on model training, explanation, and visualization allows engineers to drastically accelerate root cause analysis, reducing investigation timeframes from several days to under two hours.

**Keywords:** machine learning root cause analysis, explainable AI, lightgbm framework, shap analysis, manufacturing defect analysis, high-dimensional data processing, imbalanced dataset modeling, predictive model transparency, self-service analytics tools, correlation vs causation, non-linear data relationships, semiconductor manufacturing, global and local feature insights, python shiny deployment, human-centric machine learning

## Chapters

1. **Challenges of measuring high-precision semiconductor manufacturing components** (00:05) — Manufacturing extremely precise mirrors for lithography creates immense amounts of complex sensor and parameter data.
1. **Limitations of traditional statistical methods on high-dimensional data** (04:20) — Standard methods like linear regression struggle with non-linear relationships, missing data points, and imbalanced defect occurrence.
1. **Building user trust and finding insights via explainable AI** (06:32) — Explainable AI techniques turn black-box machine learning models into transparent tools that reveal hidden predictive patterns.
1. **Framing manufacturing root cause problems as machine learning tasks** (09:05) — Translating rising defect rates across mixed data streams into actionable modeling features enables deeper root-cause diagnostics.
1. **Handling mixed data types using the LightGBM framework** (12:00) — The LightGBM framework efficiently handles high-dimensional spaces, missing values, and non-linear interactions without extensive pre-processing.
1. **Extracting transparent feature influence using SHAP and game theory** (13:43) — Shapley additive explanations attribute influence to specific model outcomes to transform interpretability into causal pattern discovery.
1. **Identifying root causes through global and local SHAP plots** (15:29) — Global importance graphs and local dependency plots isolate specific parameter thresholds where machine defect probabilities sharply increase.
1. **Creating an internal self-service data investigation Python application** (17:58) — Combining human domain knowledge and data science into a Python toolkit enables engineers to safely conduct analyses independently.
1. **Key principles for adopting machine learning tools in engineering** (20:00) — Successful analytics tool adoption requires human-centric design, robust handling of real-world noise, and transparency regarding algorithmic limitations.

## Related Moments

- [Introduction to safety-critical machine learning in automotive contexts](https://www.wearedevelopers.com/videos/397-what-non-automotive-machine-learning-projects-can-learn-from-automotive-machine-learning-projects) (from "What non-automotive Machine Learning projects can learn from automotive Machine Learning projects")
- [Uncovering the hidden technical debt in machine learning](https://www.wearedevelopers.com/videos/392-mlops-what-s-the-deal-behind-it) (from "MLOps - What’s the deal behind it?")
- [Audience questions on practical machine learning operational strategies](https://www.wearedevelopers.com/videos/262-is-my-ai-alive-but-brain-dead-how-monitoring-can-tell-you-if-your-machine-learning-stack-is-still-performing) (from "Is my AI alive but brain-dead? How monitoring can tell you if your machine learning stack is still performing")
- [Limitations of traditional machine learning debugging approaches](https://www.wearedevelopers.com/videos/71-debugging-machine-learning-code) (from "Debugging Machine Learning Code")
- [History and growth of explainable artificial intelligence](https://www.wearedevelopers.com/videos/589-explainable-machine-learning-explained) (from "Explainable machine learning explained")
- [Embracing machine learning and predictive system behavior analytics](https://www.wearedevelopers.com/videos/874-system-resilience-surviving-the-software-storm) (from "System Resilience: Surviving the Software Storm")

## Related Articles

- [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)
- [Stephan Gillich - Bringing AI Everywhere](https://www.wearedevelopers.com/magazine/489-stephan-gillich-bringing-ai-everywhere)
- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [WWC24 Talk - Scott Hanselman - AI: Superhero or Supervillain?](https://www.wearedevelopers.com/magazine/469-wwc24-talk-scott-hanselman-ai-superhero-or-supervillain)

## Related Jobs

- [Principal Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1706410-principal-machine-learning-engineer) at **Almedia**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1597388-machine-learning-engineer) at **ZEISS Group**
- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**
- [Head of AI Applications](https://www.wearedevelopers.com/jobs/ext/1231536-head-of-ai-applications) at **ZEISS Group**