WeAreDevelopers LIVE • Nov 17, 2020

Debugging Machine Learning Code

Svetlin Penkov

Are you mistaking basic code errors for fundamental flaws in your neural network? Discover how 3D interactive debuggers and automated assertions instantly catch silent training failures.

Pause

Mute

Enter Fullscreen

#1 about 3 min

Speaker background and the need for robust models

Background of the speaker and the necessity of highly robust machine learning code in autonomous driving.

#2 about 5 min

Understanding computation and data in machine learning

How machine learning models function as computational graphs executing mathematical operations across multi-dimensional data arrays.

#3 about 6 min

The two interconnected loops of model development

The distinction between analyzing structural semantic flaws and handling purely mathematical runtime performance issues.

#4 about 12 min

Limitations of traditional machine learning debugging approaches

Why mental verification, terminal printouts, simple plotting, and custom dashboards fail to scale for complex architectures.

#5 about 18 min

Introducing the FMRI debugger for 3D computational insights

How the FMRI debugger visually maps out code, data, gradients, and automated assertions natively in a fully interactive spatial interface.

#6 about 7 min

Interactive demo of visual tensor inspection and assertions

Resolving vanishing gradients and invalid probability inputs by visually inspecting multi-dimensional structures alongside automated mathematical tests.

#7 about 3 min

Visualizing large-scale model architectures like ResNet-101

Expanding deep individual computational layers and verifying feature maps within large open-source image recognition architectures.

Matching moments

1:31 min

Debugging distributed systems with artificial intelligence tools

5:01 min

Leveraging large language models for code optimization and development

Stephan Gillich

Stephan Gillich +3 · World Congress 2024

2:41 min

Leveraging AI tooling for accelerated learning and productivity

Dan Mindru · Coffee With Developers

2:16 min

Interpreting complex model behavior without ground truth data

Kilian Kluge +1 · World Congress 2022

1:51 min

Debugging costs of runtime data shape mismatches

Violina Popova

Violina Popova · Europe 2026 Virtual

3:01 min

Balancing artificial intelligence tools with foundational software engineering skills

Tim Ruscica · Coffee With Developers

More from WeAreDevelopers LIVE

WeAreDevelopers LIVE - Is CSS Catching Up to Flash? thumbnail

WeAreDevelopers LIVE - Is CSS Catching Up to Flash?

Chris Heilmann, Daniel Cranney, Bramus Van Damme

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along thumbnail

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along

Chris Heilmann, Daniel Cranney, Krzysztof Cieślak

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak thumbnail

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Open Source Agent Toolkits thumbnail

WeAreDevelopers LIVE - Open Source Agent Toolkits

Chris Heilmann, Daniel Cranney, Eddie Jaoude

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude thumbnail

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude

Chris Heilmann, Daniel Cranney

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

Upcoming sessions on this topic

World Congress 2026 North America

September 25, 2026 · 16:50–17:20

Stage 7

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan

World Congress 2026 North America

September 25, 2026 · 14:50–15:20

Stage 2

State of the Software Factory

Dexter Horthy

Co-Founder of HumanLayer

Dexter Horthy

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 1

Anatomy of an AI Request: Where Latency and Cost Are Really Born

Dan Fu

VP of Kernels at Together AI

Dan Fu

World Congress 2026 North America

September 25, 2026 · 13:30–14:00

Mainstage

A Hands-On Developer Guide to Inference Engineering

Ankit Patel, Philip Kiely

Ankit Patel

Philip Kiely

World Congress 2026 North America

September 24, 2026 · 16:10–16:40

Stage 3

Sandboxing the Swarm: Building Secure, Serverless AI Agents with Wasm

Thorsten Hans

Sr. Developer Advocate @ Akamai Technologies

Thorsten Hans

World Congress 2026 North America

September 25, 2026 · 11:00–11:30

Stage 6

Loop Engineering: Designing and Observing the Loops That Run Your Agents With Opik

Abigail Morgan

AI Developer Advocate at Comet

Abigail Morgan

Related videos

Getting Started with Machine Learning thumbnail

LIVE • almost 5 years ago

Getting Started with Machine Learning

Alexandra Waldherr

Why LLMs Need Observability and How to Do It thumbnail

Europe 2026 Virtual • 3 months ago

Why LLMs Need Observability and How to Do It

Rahul Gupta

Talk to the Duck - Secrets of the Debugging Masters thumbnail

World Congress 2022 • over 4 years ago

Talk to the Duck - Secrets of the Debugging Masters

Shai Almog

Debugging Unveiled: Exploring Debugger Internals and Hidden Gems thumbnail

LIVE • over 2 years ago

Debugging Unveiled: Exploring Debugger Internals and Hidden Gems

Johannes Bechberger

Building Your Own Classification Model with JavaScript - Coffee with Developers - Carly Richmond thumbnail

Coffee With Developers • over 1 year ago

Building Your Own Classification Model with JavaScript - Coffee with Developers - Carly Richmond

Carly Richmnd

A hundred ways to wreck your AI - the (in)security of machine learning systems thumbnail

World Congress 2023 • almost 3 years ago

A hundred ways to wreck your AI - the (in)security of machine learning systems

Balázs Kiss

Related articles

DC

Daniel Cranney

AI-Powered Debugging: The Future of Fixing Your Code

The reality of being a developer is that we often spend more time fixing code that actually writing new code. Debugging is such a core and time-consuming part of software development, it’s no surpr...

over 1 year ago

A page of code with a magnifying glass over it, highlighting the word BUG in red text.

BB

Benedikt Bischof

MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production

Welcome to this issue of the WeAreDevelopers Live Talk series. This article recaps an interesting talk by Bas Geerdink who gave advice on MLOps.‍About the speaker:‍Bas is a programmer, scientist, a...

about 4 years ago

MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production

BB

Benedikt Bischof

MLOps And AI Driven Development

Welcome to this issue of the WeAreDevelopers Dev Talk Recap series. This article recaps an interesting talk by Natalie Pistunovic who spoke about the development of AI and MLOps. What you will lear...

over 4 years ago

MLOps And AI Driven Development

BB

Benedikt Bischof

MLOps – What’s the deal behind it?

Welcome to this issue of the WeAreDevelopers Live Talk series. This article recaps an interesting talk by Nico Axtmann who introduced us to MLOpsAbout the speaker:Nico Axtmann is a seasoned machine...

almost 4 years ago

MLOps – What’s the deal behind it?

From learning to earning

Jobs that call for the skills explored in this talk.

about 1 month ago • Verified

Distributed Training and Inference Engineer

San Francisco, United States

Expert $190k–250k Linux kernel

about 2 months ago

MLOps AI Engineer

TeamViewer Germany GmbH,

Austin, TX, United States

Expert Caching Routing Standard Sql

Principal Software Engineer, AI Inference Runtime

Seattle, WA, United States

Expert $262k Compilers Low Latency Concurrency

about 1 month ago • Verified

LLM Training Engineer

San Francisco, United States

Expert $155k–220k Python

Principal Software Engineer, AI Inference Cloud

Seattle, WA, United States

Expert $262k Pytorch TensorRT Kubernetes

about 1 month ago • Verified

Model Implementation Engineer

San Francisco, United States

Expert $165k–220k Python