> Markdown version of [/videos/1217-can-machines-dream-of-secure-code-emerging-ai-security-risks-in-llm-driven-developer-tools?t=548](https://www.wearedevelopers.com/videos/1217-can-machines-dream-of-secure-code-emerging-ai-security-risks-in-llm-driven-developer-tools?t=548). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Can Machines Dream of Secure Code? Emerging AI Security Risks in LLM-driven Developer Tools

Is your AI coding assistant quietly introducing critical vulnerabilities? Discover how LLM tools hallucinate insecure packages and learn why you must treat auto-completed code like untrusted user input.

- **Speakers:** [Liran Tal](https://www.wearedevelopers.com/@liran-tal)
- **Event:** WeAreDevelopers LIVE
- **Published:** September 25, 2024
- **Duration:** 35:37
- **URL:** https://www.wearedevelopers.com/videos/1217-can-machines-dream-of-secure-code-emerging-ai-security-risks-in-llm-driven-developer-tools

## Summary

Software development has evolved from writing original logic to copying from forums, importing open-source packages, and now relying entirely on AI coding assistants. This shift means a developer's code is often no longer their own, bringing new supply chain and security risks. As developers fixate on shipping product logic, they risk their "vulnerable line of code" becoming their "first security regret," especially when adopting AI tools that prioritize functional output over secure practices.

LLM-driven developer tools inherently learn from vast datasets, meaning low-confidence malicious data can easily be absorbed as high-confidence trusted data. The risks manifest through AI hallucinating non-existent software packages—opening the door for supply chain traps—or recommending naive shell executions that invite command injections. Furthermore, AI often lacks context-aware reasoning. For example, suggesting a generic HTML escaping function might successfully neutralize a DOM injection but fail completely when applied to an image attribute, leaving the application exposed to cross-site scripting (XSS). Even worse, AI tools actively scan local developer environments and amplify existing bad practices, perfectly mirroring nearby SQL injection vulnerabilities into newly auto-completed code.

Because machines undeniably dream of insecure code, developers must adapt how they interact with AI companions. Relying on an LLM to validate logic is insufficient if foundational prompt injection risks and path traversal flaws remain unchecked. To counterbalance these emerging threats, teams should treat AI-generated code exactly like untrusted user input, scrutinizing it meticulously via peer reviews and referencing standards like the OWASP Top 10 for LLMs. Ultimately, integrating static application security testing (SAST) and "secure on save" tooling directly into the IDE ensures that while coding assistance accelerates development, automated guardrails securely anchor it.

**Keywords:** ai coding assistants, llm security risks, prompt injection, path traversal vulnerabilities, command execution risks, nosql injection, ai hallucination vulnerabilities, software supply chain defense, cross-site scripting, owasp top 10 for llms, static application security testing, insecure code amplification, ide security extensions, threat modeling ai output, context-aware input sanitization

## Chapters

1. **Understanding NoSQL injection in basic middleware** (00:00) — How unvalidated user input flows into sensitive methods to cause NoSQL injections.
1. **Analyzing a real-world NoSQL vulnerability in Rocket Chat** (03:51) — Reviewing a widely deployed NoSQL injection affecting password reset policies.
1. **The evolution of code reuse and dependency risks** (06:28) — Tracing how developers adopt external code from forums to package registries and AI assistants.
1. **Core AI security risks and data poisoning** (09:08) — Why trusting language model outputs equates to trusting potentially poisoned training data.
1. **Exploiting exposed language models in commercial applications** (12:26) — How integrating conversational interfaces into products can accidentally expose internal operations.
1. **Path traversal risks in AI-generated file uploads** (14:44) — Discovering hidden path traversal flaws when copying generated file upload code directly into production.
1. **Command injection vulnerabilities in ChatGPT snippets** (19:00) — How generative models encourage risky operations like shell execution without adequate security warnings.
1. **Exploiting hallucinated software packages and package dependencies** (22:18) — The security implications of models recommending phantom packages that attackers can register maliciously.
1. **Path traversal vulnerabilities built by IDE assistants** (24:51) — Reviewing explicit file system access flaws created by typical code autocomplete suggestions.
1. **Cross-site scripting via context-blind HTML escaping** (27:15) — Why generalized sanitization methods fail against context-dependent injection constraints like HTML element attributes.
1. **Prompt injection and data exfiltration inside IDEs** (31:25) — How malicious repository comments manipulate coding assistants to leak sensitive developer data.
1. **How coding assistants amplify existing insecure code** (33:00) — Why autocomplete tools replicate and spread legacy SQL vulnerabilities throughout active projects.
1. **Mitigating AI code risks and OWASP recommendations** (34:28) — Utilizing automated scanning tools and security standards to validate machine-generated logic.

## Related Moments

- [The impact and risks of AI generated code](https://www.wearedevelopers.com/videos/1280-navigating-the-future-of-junior-developers-in-tech) (from "Navigating the Future of Junior Developers in Tech")
- [Top security vulnerabilities for AI applications](https://www.wearedevelopers.com/videos/1637-delay-the-ai-overlords-how-oauth-and-openfga-can-keep-your-ai-agents-from-going-rogue) (from "Delay the AI Overlords: How OAuth and OpenFGA Can Keep Your AI Agents from Going Rogue")
- [Managing security risks in AI-accelerated development processes](https://www.wearedevelopers.com/videos/100323-automated-security-for-the-entire-sdlc) (from "Automated Security for the Entire SDLC")
- [Security vulnerabilities introduced by frictionless AI code generation](https://www.wearedevelopers.com/videos/1267-exploring-ai-opportunities-and-risks-in-development) (from "Exploring AI: Opportunities and Risks in Development")
- [Risks of malicious VS Code extensions and AI assistants](https://www.wearedevelopers.com/videos/1794-wearedevelopers-live-from-javascript-to-webassembly-high-performance-charting-and-more) (from "WeAreDevelopers LIVE – From JavaScript to WebAssembly, High-Performance Charting and More")
- [Analyzing software composition risks and shadow AI vulnerabilities](https://www.wearedevelopers.com/videos/100248-reporting-active-exploits-in-24-hours-are-you-ready-for-the-cra) (from "Reporting Active Exploits in 24 Hours: Are You Ready for the CRA?")

## Related Articles

- [Exploring AI: Opportunities and Risks for Developers](https://www.wearedevelopers.com/magazine/522-exploring-ai-opportunities-and-risks-for-developers)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [Transforming Software Development: The Role of AI and Developer Tools](https://www.wearedevelopers.com/magazine/527-transforming-software-development-the-role-of-ai-and-developer-tools)
- [One billion (bad?) developers: How AI is changing the way we learn to code](https://www.wearedevelopers.com/magazine/516-one-billion-bad-developers-how-ai-is-changing-the-way-we-learn-to-code)

## Related Jobs

- [Staff Developer Advocate, GitHub Security Lab](https://www.wearedevelopers.com/jobs/ext/1921051-staff-developer-advocate-github-security-lab) at **GitHub**
- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**
- [Engineer, Offensive Security Organization](https://www.wearedevelopers.com/jobs/ext/1992296-engineer-offensive-security-organization) at **Twilio**
- [Security Architect - AI](https://www.wearedevelopers.com/jobs/ext/1581899-security-architect-ai) at **ZEISS Group**
- [Principal Software Engineer, Enterprise AI Platform](https://www.wearedevelopers.com/jobs/ext/1467292-principal-software-engineer-enterprise-ai-platform) at **GitHub**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**