> Markdown version of [/videos/2131-evals-vs-evil-ai-and-package-security-laurie-voss?t=650](https://www.wearedevelopers.com/videos/2131-evals-vs-evil-ai-and-package-security-laurie-voss?t=650). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Evals vs. Evil - AI and Package Security - Laurie Voss

Laurie Voss warns that unchecked AI agents are poisoning open-source supply chains. Learn why entry-level coding is dead, and how strict spec writing will save your architecture.

- **Speakers:** Laurie Voss
- **Event:** Coffee With Developers
- **Published:** July 20, 2026
- **Duration:** 16:57
- **URL:** https://www.wearedevelopers.com/videos/2131-evals-vs-evil-ai-and-package-security-laurie-voss

## Summary

The rapid rise of AI coding assistants and agentic workflows has forced development teams to rethink how they test and validate code. In the machine learning engineering space, "evals" serve as the equivalent of unit tests for large language models (LLMs). Because output is non-deterministic, traditional string matching fails. Instead, developers must use a secondary model to judge the primary output—ideally utilizing models from different research labs to prevent self-preference bias, while adopting cheaper open-source models to control token pricing.

As the economic realities of running AI models temper the initial hype, the industry faces a growing tidal wave of "AI slop," particularly within open-source repositories overwhelmed by automated pull requests. This unchecked velocity introduces severe vulnerabilities into software supply chains. Security defenses and default guardrails in package managers like NPM currently lag behind the agentic tools used by attackers to find exploits. While alternative clients like pnpm offer safer defaults, the broader ecosystem must embrace stricter boundaries to safely sustain the high deployment speeds enabled by model-generated code.

The commoditization of raw code generation is fundamentally altering software engineering career trajectories. The traditional entry-level role of writing boilerplate code is disappearing, replaced by AI agents taking instruction. Consequently, developers must skip foundational manual coding roles and focus directly on systems architecture and rigorous spec writing. In an AI-native development environment, crafting a voluminous, highly detailed specification has become the critical skill required to corral an LLM into producing high-quality, structured application logic.

**Keywords:** machine learning evals, llm unit testing protocols, ai self-preference bias, token cost optimization, open-source ai models, ai generated code slop, automated pull request spam, software supply chain attacks, npm vulnerability exploitation, pnpm security defaults, agentic ai workflows, ai-native software architecture, specification writing skills, developer job market trends, non-deterministic output validation

## Chapters

1. **Defining and implementing LLM evaluation strategies** (00:02) — Because traditional unit tests fail on variable LLM outputs, utilizing a separate model for evaluation serves as an effective testing substitute.
1. **Managing AI costs with open-source models** (02:04) — To mitigate volatile per-token subscription expenses, engineering teams can adopt open-source models that provide comparable performance at a lower cost.
1. **Assessing the real value of the agentic movement** (03:42) — While the agentic tech bubble persists, foundational models offer practical value by reliably converting unstructured data into structured pipelines.
1. **Combating AI slop in open-source projects** (05:50) — The careless misuse of AI generation tools is actively overwhelming open-source maintainers with a flood of low-quality, automated pull requests.
1. **Addressing supply chain attacks in package managers** (08:27) — To combat rising supply chain attacks fueled by AI-generated code, ecosystems must update default client configurations to prioritize repository security.
1. **Balancing developer velocity with vital security guardrails** (10:50) — As code generation tools drastically lower implementation costs, organizations must introduce robust automated guardrails to sustainably enable higher deployment velocities.
1. **Redefining developer roles and software architecture requirements** (13:08) — Since AI agents can now handle basic coding tasks, early-career engineers must pivot toward mastering software architecture and complex spec writing.
1. **The enduring value of in-person developer events** (15:27) — Despite the rise of remote tools, physical proximity and spontaneous human connections remain critical for cross-domain networking and career growth.

## Related Moments

- [Shifting developer workloads and realistic AI productivity gains](https://www.wearedevelopers.com/videos/1830-wearedevelopers-live-speculaitions) (from "WeAreDevelopers LIVE - SpeculAItions")
- [Security integration and AI skepticism in developer tooling](https://www.wearedevelopers.com/videos/1830-wearedevelopers-live-speculaitions) (from "WeAreDevelopers LIVE - SpeculAItions")
- [Motivations for adopting AI to enhance developer productivity](https://www.wearedevelopers.com/videos/1266-navigating-the-ai-revolution-in-software-development) (from "Navigating the AI Revolution in Software Development")
- [The impact and risks of AI generated code](https://www.wearedevelopers.com/videos/1280-navigating-the-future-of-junior-developers-in-tech) (from "Navigating the Future of Junior Developers in Tech")
- [Balancing AI tool mandates with developer trust and productivity](https://www.wearedevelopers.com/videos/1365-wearedevelopers-live-the-weekly-developer-show-with-chris-heilmann-and-daniel-cranney) (from " WeAreDevelopers LIVE - the weekly developer show with Chris Heilmann and Daniel Cranney")
- [Navigating developer bottlenecks and human accountability](https://www.wearedevelopers.com/videos/100265-fireside-chat-in-conversation-with-werner-vogels-cto-of-amazon-com) (from "Fireside Chat - In conversation with Werner Vogels, CTO of Amazon.com")

## Related Articles

- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [Exploring AI: Opportunities and Risks for Developers](https://www.wearedevelopers.com/magazine/522-exploring-ai-opportunities-and-risks-for-developers)
- [What is Software Engineering in the Age of AI?](https://www.wearedevelopers.com/magazine/640-what-is-software-engineering-in-the-age-of-ai)
- [What is Agentic Programming and Why Should Developers Care?](https://www.wearedevelopers.com/magazine/625-what-is-agentic-programming-and-why-should-developers-care)

## Related Jobs

- [Staff Developer Advocate, GitHub Security Lab](https://www.wearedevelopers.com/jobs/ext/1921051-staff-developer-advocate-github-security-lab) at **GitHub**
- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Principal Software Engineer, Enterprise AI Platform](https://www.wearedevelopers.com/jobs/ext/1467292-principal-software-engineer-enterprise-ai-platform) at **GitHub**
- [Senior AI Agent Software Engineer (Go, Python) (m/f/x)](https://www.wearedevelopers.com/jobs/48277-senior-ai-agent-software-engineer-go-python-m-f-x) at **Dynatrace**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/588393-machine-learning-engineer) at **Twilio**