> Markdown version of [/videos/519-finding-the-unknown-unknowns-intelligent-data-collection-for-autonomous-driving-development?t=868](https://www.wearedevelopers.com/videos/519-finding-the-unknown-unknowns-intelligent-data-collection-for-autonomous-driving-development?t=868). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Finding the unknown unknowns: intelligent data collection for autonomous driving development

How do self-driving systems learn from unpredictable anomalies without uploading terabytes of junk data? Discover how intelligent edge computing captures only the critical unknown unknowns for rapid model retraining.

- **Speakers:** Liang Yu
- **Event:** World Congress 2022
- **Published:** June 15, 2022
- **Duration:** 19:24
- **URL:** https://www.wearedevelopers.com/videos/519-finding-the-unknown-unknowns-intelligent-data-collection-for-autonomous-driving-development

## Summary

Autonomous driving systems face a significant "long-tail" problem, frequently encountering unpredictable or novel "unknown unknowns" that lead to dangerous perception failures. Brute-force data acquisition to capture these corner cases is economically unfeasible and yields massive amounts of irrelevant junk data. To solve this, Volkswagen subsidiary Cariad developed the "Big Loop" data aggregation system and "Instinct" (Intelligent Data Collector), which utilize edge computing to dramatically reduce data upload volumes from thousands of useless frames to just a handful of high-value edge cases.

Running directly on a vehicle's embedded hardware—specifically within an isolated "protected area blade" to guarantee passenger safety—Instinct evaluates perception models in real time by calculating component-wise uncertainty scores. For example, if a model outputs high uncertainty bounds when segmenting an ambiguous traffic light or encountering a novel object, the system selectively triggers an upload. This shifts the computational burden to the intelligent edge, curating only the precise data points necessary to resolve model confusion.

By leveraging this curated data for human or automated labeling, engineering teams can rapidly retrain neural networks to overcome distribution shifts without building siloed, regional models. The retrained model representation is then safely pushed back to the vehicle via an over-the-air (OTA) update in a matter of minutes. This automated, active learning pipeline drives continuous data-driven development, optimizing cloud storage, iterative engineering costs, and the ongoing safety of self-driving perception systems.

**Keywords:** autonomous driving, data acquisition, unknown unknowns, edge computing, anomaly detection, uncertainty scoring, active learning, long-tail problem, over-the-air updates, bayesian inference, perception models, distribution shift, embedded machine learning, continuous data-driven development

## Chapters

1. **Unified tech stack and hardware for Volkswagen** (00:05) — An introduction to CARIAD's unified tech stack and hardware platform designed to accelerate innovation for autonomous driving functions.
1. **Traditional sensor data acquisition and storage challenges** (01:58) — The limitations of locally recording all sensor data for manual transfer and inefficient storage.
1. **Introducing the Big Loop intelligent data system** (03:02) — How the Big Loop uses isolated hardware components to safely test software and filter junk information.
1. **Identifying unknown unknowns in perception model scenarios** (04:56) — Defining corner cases like distribution shifts and rare objects that cause perception models to fail on the road.
1. **Methods for finding out-of-distribution perception scenarios** (06:45) — Comparing bayesian inference, anomaly detection, and ensemble methods for finding out-of-distribution samples.
1. **Filtering frames using uncertainty scores in Instinct** (07:36) — How the Instinct software calculates component-wise uncertainty scores on segmentation output to trigger intelligent data uploads.
1. **Real-world retraining and over-the-air model deployment** (09:42) — A pipeline demonstrating how models fail, capture triggers, retrain with auto-labeling, and deploy via over-the-air updates.
1. **Scaling the pioneering test fleet for data collection** (13:17) — Retrofitting existing vehicles enables continuous active learning and data-driven development ahead of standard deployment.
1. **Audience Q&A on autonomous driving models and data** (14:28) — Audience discussions cover handling critical safety decisions, local edge execution, dynamic uncertainty thresholds, and global training datasets.

## Related Moments

- [Managing synthetic data unpredictability alongside multi-sensor autonomous vehicle deployment](https://www.wearedevelopers.com/videos/100039-physical-ai-for-the-next-wave-of-industrial-digitalisation) (from "Physical AI for the Next Wave of Industrial Digitalisation")
- [Enhancing automated driving through swarm data processing](https://www.wearedevelopers.com/videos/1114-software-defines-the-vehicle-why-customers-and-developers-will-love-cars-even-more) (from "Software defines the vehicle: Why customers and developers will love cars even more")
- [Audience questions on data integration and future action models](https://www.wearedevelopers.com/videos/899-creating-industry-ready-solutions-with-llm-models) (from "Creating Industry ready solutions with LLM Models")
- [Speaker background and the need for robust models](https://www.wearedevelopers.com/videos/71-debugging-machine-learning-code) (from "Debugging Machine Learning Code")
- [Addressing participant questions on liability and machine learning](https://www.wearedevelopers.com/videos/374-the-future-of-automotive-mobility-upcoming-e-e-architectures-v2x-and-its-challenges) (from "The future of automotive mobility: Upcoming E/E architectures, V2X and its challenges")
- [Creating virtual feedback cycles for in-car artificial intelligence platforms](https://www.wearedevelopers.com/videos/397-what-non-automotive-machine-learning-projects-can-learn-from-automotive-machine-learning-projects) (from "What non-automotive Machine Learning projects can learn from automotive Machine Learning projects")

## Related Articles

- [How software is steering vehicle technology](https://www.wearedevelopers.com/magazine/515-how-software-is-steering-vehicle-technology)
- [Stephan Gillich - Bringing AI Everywhere](https://www.wearedevelopers.com/magazine/489-stephan-gillich-bringing-ai-everywhere)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)

## Related Jobs

- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/381484-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/353953-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/319507-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**