> Markdown version of [/videos/2110-the-algorithm-that-nearly-killed-me-when-testing-isn-t-enough](https://www.wearedevelopers.com/videos/2110-the-algorithm-that-nearly-killed-me-when-testing-isn-t-enough). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# The Algorithm That Nearly Killed Me: When Testing Isn't Enough

Senior engineer Andrew Paul's insulin pump nearly killed him despite perfectly passing tests. Discover how to use property-based testing to catch chaotic edge cases before they become catastrophic.

- **Speakers:** [Andrew Paul](https://www.wearedevelopers.com/@andrew-paul)
- **Event:** World Congress 2026 Europe - Virtual Stage
- **Published:** July 3, 2026
- **Duration:** 47:19
- **URL:** https://www.wearedevelopers.com/videos/2110-the-algorithm-that-nearly-killed-me-when-testing-isn-t-enough

## Summary

An insulin pump algorithm designed to manage diabetes nearly triggered a fatal medical emergency, not because the code lacked tests, but because it confidently learned the wrong patterns. Through this harrowing personal experience, senior software engineer Andrew Paul illustrates a critical failure in modern software development: the dangerous gap between passing tests and actual user safety. While traditional unit and integration tests validate individual rules, they often fail to capture the "edges between the edges"—the chaotic intersections where independently safe rules interact to create life-threatening or system-destroying outcomes.

This testing blind spot is no longer confined to medical devices. As the industry shifts from processing data to making autonomous decisions with machine learning, algorithms adapt and discover unforeseen patterns well outside developer intent. By distributing quality assurance responsibilities across teams without equipping them with specialized expertise, organizations are lulled into a false sense of security governed by green CI builds and arbitrary coverage thresholds. We broadly test for successful implementation but consistently neglect to test for catastrophic impact, echoing historical industry failures where autonomous systems were blindly trusted despite being systematically wrong.

To build software worthy of genuine trust, engineering teams must rapidly overhaul their testing paradigms for complex systems. This involves mapping theoretical feature collisions via interaction matrices and deploying property-based testing libraries like Hypothesis, FastCheck, or QuickCheck to programmatically generate boundary scenarios. Teams should also adapt chaos engineering to stress-test business logic, explicitly document "must-never invariants" that trigger loud pipeline failures, and monitor decision boundaries in machine learning models rather than treating accuracy as the sole metric. Ultimately, engineers deploying high-stakes code must ask an uncomfortable final question: would you genuinely trust your own system with your health, money, or freedom?

**Keywords:** autonomous system validation, edge case intersections, software impact testing, property-based testing frameworks, chaos engineering business logic, machine learning decision boundaries, must-never invariants, business logic observability, feature flag rule interactions, threat modeling boundaries, ai pattern recognition risks, interaction matrices, test-driven development failures, software user safety

## Chapters

1. **Personal background and biological management metrics** (00:02) — Maintaining tight biological control through continuous monitoring dictates how specialized medical metrics are evaluated.
1. **System setup and early automation warnings** (05:53) — A highly regulated closed-loop insulin pump introduces immediate vendor red flags before temporarily stabilizing during initial software deployment.
1. **Algorithm failure against diabetic medical boundaries** (11:43) — A life-threatening incident reveals the integration flaws between application instructions and mechanical delivery that almost caused ketoacidosis.
1. **Testing the edges between intersecting edge cases** (17:58) — Standard rules passing isolated tests can trigger dangerous software behavior when multiple edge cases intersect unexpectedly.
1. **Machine learning adaptation and the illusion of safety** (23:18) — Artificial intelligence adapts to independent patterns and introduces extreme risk when unit tests only confirm base implementation.
1. **Evaluating software impact instead of isolated implementation** (26:23) — The Post Office Horizon scandal illustrates why robust test coverage means nothing if software systematically destroys human lives.
1. **Distributing quality accountability across cross-functional modern teams** (29:36) — Removing dedicated quality assurance testers requires equipping broad engineering teams with rigorous analytical skills to evaluate complex user risk.
1. **Applying advanced testing strategies for complex business rules** (31:06) — Practical applications of interaction matrices, property-based tests, and chaos engineering ensure strict adherence to core system invariants.
1. **Deploying observability and dynamic controls against unexpected logic** (37:07) — Structured validation of algorithmic decision boundaries, granular telemetry observability, and dynamic feature flags integrate directly with application threat modeling.
1. **Fostering empathy and accountability in engineering decisions** (42:11) — Understanding user environments allows engineering teams to accurately assess whether autonomous systems truly earn high-stakes human trust.
1. **Closing argument for prioritizing definitive user safety** (44:23) — Intentional transparency embedded into automated processes enforces strict limits and judges software success entirely by user preservation.

## Related Moments

- [Preventing catastrophic failures in high-stakes software](https://www.wearedevelopers.com/videos/720-when-testing-just-doesn-t-cut-it) (from "When testing just doesn’t cut it")
- [Real-world tech failures from missing ethical guardrails](https://www.wearedevelopers.com/videos/1569-kill-switch-or-moral-compass-who-programs-ai-s-conscience) (from "Kill Switch or Moral Compass: Who Programs AI’s Conscience?")
- [Balancing technical fail-safes with proactive moral compasses](https://www.wearedevelopers.com/videos/1569-kill-switch-or-moral-compass-who-programs-ai-s-conscience) (from "Kill Switch or Moral Compass: Who Programs AI’s Conscience?")
- [Preventing severe failures through rigorous QA and organizational safeguards](https://www.wearedevelopers.com/videos/423-the-software-bug-all-stars-and-what-we-can-learn-from-them) (from "The Software Bug All Stars - and what we can learn from them")
- [Mitigating cascading failures in AI generated software environments](https://www.wearedevelopers.com/videos/100242-running-ai-at-scale-the-secret-ingredients) (from "Running AI at Scale: The Secret Ingredients")
- [Navigating developer bottlenecks and human accountability](https://www.wearedevelopers.com/videos/100265-fireside-chat-in-conversation-with-werner-vogels-cto-of-amazon-com) (from "Fireside Chat - In conversation with Werner Vogels, CTO of Amazon.com")

## Related Articles

- [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)
- [What is Software Engineering in the Age of AI?](https://www.wearedevelopers.com/magazine/640-what-is-software-engineering-in-the-age-of-ai)
- [Never delegate the understanding](https://www.wearedevelopers.com/magazine/749-never-delegate-the-understanding)
- [Exploring AI: Opportunities and Risks for Developers](https://www.wearedevelopers.com/magazine/522-exploring-ai-opportunities-and-risks-for-developers)

## Related Jobs

- [Principal Software Engineer, Enterprise AI Platform](https://www.wearedevelopers.com/jobs/ext/1467292-principal-software-engineer-enterprise-ai-platform) at **GitHub**
- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**
- [Senior Engineer, Infrastructure Platform](https://www.wearedevelopers.com/jobs/ext/328836-senior-engineer-infrastructure-platform) at **Intercom, Inc.**
- [Engineer, Offensive Security Organization](https://www.wearedevelopers.com/jobs/ext/1992296-engineer-offensive-security-organization) at **Twilio**
- [Staff Developer Advocate, GitHub Security Lab](https://www.wearedevelopers.com/jobs/ext/1921051-staff-developer-advocate-github-security-lab) at **GitHub**
- [Principal Software Engineer, Identity](https://www.wearedevelopers.com/jobs/ext/1469181-principal-software-engineer-identity) at **GitHub**