> Markdown version of [/videos/986-manipulating-the-machine-prompt-injections-and-counter-measures](https://www.wearedevelopers.com/videos/986-manipulating-the-machine-prompt-injections-and-counter-measures). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Manipulating The Machine: Prompt Injections And Counter Measures

Hackers are tricking AI chatbots into selling cars for $1 and stealing 2FA tokens. Learn why complex system prompts fail and how to properly defend against prompt injections.

- **Speakers:** [Georg](https://www.wearedevelopers.com/@georg)
- **Event:** World Congress 2024
- **Published:** August 20, 2024
- **Duration:** 27:10
- **URL:** https://www.wearedevelopers.com/videos/986-manipulating-the-machine-prompt-injections-and-counter-measures

## Summary

This presentation unpacks the modern security landscape of large language models, focusing specifically on the anatomy and countermeasures of prompt injections and prompt stealing. By examining the prompt stack—comprising system instructions, contextual data, and unchecked user input—developers can see how feeding unfiltered text into an AI mirrors the dangers of SQL injection and cross-site scripting. Real-world case studies illustrate the severity of these vulnerabilities, from manipulating a car dealership chatbot into offering a vehicle for $1, to bypassing automated hr evaluation platforms by hiding 'perfect candidate' instructions within a submitted CV, to exploiting ai-powered email summarizers to secretly forward intercepted two-factor authentication tokens. To defend against these exploits, trying to build impenetrable system prompts is inherently flawed; constantly adding restrictive rules merely generates complex 'spaghetti prompts' that are effortlessly bypassed with basic override commands. Instead, robust architecture requires treating all system instructions as public data and stripping sensitive business secrets or writable api access from the model entirely. For production environments, engineers are advised to transition toward fine-tuned models that rely less on hackable instructions and to implement adversarial prompt detectors like nemo guardrails to actively filter malicious user inputs before they reach the core system.

**Keywords:** prompt injection countermeasures, prompt stealing attacks, system prompt manipulation, malicious user input handling, retrieval augmented generation, automated hr evaluation bypass, ai chatbot exploitation, llm api vulnerability, spaghetti prompts, fine-tuned model security, adversarial prompt detectors, nemo guardrails, conversational ai security rules

## Chapters

1. **Anatomy of large language model prompts and structural layers** (01:32) — The three distinct layers of user input, context, and system instructions shape the fundamental behavior of generative language models.
1. **Understanding the risks of unchecked user inputs in chatbots** (05:02) — Unfiltered prompts leave applications vulnerable to unintended commands and logic overrides remarkably similar to traditional database injection attacks.
1. **Mitigating basic prompt manipulation attempts using explicit system rules** (06:48) — Adding concrete semantic limits and behavioral boundaries in the system prompt block filters out naive and rudimentary adversarial commands.
1. **Bypassing system constraints by extracting hidden instructions and rules** (08:08) — Attackers leverage conversational logic conventions to force applications into improperly revealing internal instruction sets directly back to the user.
1. **Extracting confidential business information from configured knowledge bases** (11:47) — Manipulated user inputs can subvert logic flows to silently expose proprietary backend files originally intended solely for context enhancement.
1. **Manipulating automated talent evaluation systems with hidden text injections** (13:34) — Embedding invisible instructions into resumes deliberately forces automated screening pipelines to render favorable assessments without exhaustive human oversight.
1. **Exploiting integrated application programming interfaces for unauthorized data extraction** (14:59) — Deliberately crafted external messages reliably hijack autonomous integration processes to systematically intercept authentication tokens sent to an inbox.
1. **Establishing foundational limitations to reduce system prompt injection vulnerabilities** (17:25) — Restricting integrations exclusively to read-only capabilities drastically minimizes potential exploitation vectors by neutralizing external write permissions.
1. **Defending against targeted prompt attacks by deploying fine-tuned models** (21:11) — Training isolated learning environments directly on specialized domain data drastically diminishes inherent susceptibility to adversarial instructional overrides.
1. **Filtering malicious application inputs utilizing adversarial prompt detection classifiers** (22:28) — Secondary language model deployments running pattern recognition databases act as robust firewalls by classifying problematic submissions prior to evaluation.
1. **Acknowledging current limitations in securing generative artificial intelligence applications** (24:57) — Since zero operational guarantees currently exist avoiding application compromises, administrators should practically consider all system text publicly viewable information.

## Related Moments

- [Prompt injection vulnerabilities and contextual mitigation testing challenges](https://www.wearedevelopers.com/videos/824-hacking-ai-how-attackers-impose-their-will-on-ai) (from "Hacking AI - how attackers impose their will on AI")
- [Bypassing language model safeguards utilizing contextual prompt injection attacks](https://www.wearedevelopers.com/videos/1286-wearedevelopers-live-browser-extensions-honey-scam-jailbreaking-llms-and-more) (from "WeAreDevelopers Live: Browser Extensions, Honey Scam, Jailbreaking LLMs and more")
- [Understanding AI chatbot vulnerabilities and stateful attacks](https://www.wearedevelopers.com/videos/100300-testing-ai-agents-automated-evaluation-for-chatbots-rag-systems) (from "Testing AI Agents: Automated Evaluation for Chatbots & RAG Systems")
- [Prompt injections bypassing language model service guardrails](https://www.wearedevelopers.com/videos/100115-dangerous-reactivity-why-ai-output-is-the-new-xss) (from "Dangerous Reactivity: Why AI Output Is the New XSS")
- [Prompt engineering techniques and security vulnerabilities](https://www.wearedevelopers.com/videos/1222-devops-for-ai-running-llms-in-production-with-kubernetes-and-kubeflow) (from "DevOps for AI: running LLMs in production with Kubernetes and KubeFlow")
- [Security risks involving prompt injection attacks](https://www.wearedevelopers.com/videos/624-the-shadows-that-follow-the-ai-generative-models) (from "The shadows that follow the AI generative models")

## Related Articles

- [WWC24 Talk - Scott Hanselman - AI: Superhero or Supervillain?](https://www.wearedevelopers.com/magazine/469-wwc24-talk-scott-hanselman-ai-superhero-or-supervillain)
- [How to Bypass ChatGPT’s Filter With Examples](https://www.wearedevelopers.com/magazine/332-how-to-bypass-chatgpt-s-filter-with-examples)
- [Prompt Engineering is a Job of the Past](https://www.wearedevelopers.com/magazine/342-prompt-engineering-is-a-job-of-the-past)
- [The Prompt Engineer ✍️](https://www.wearedevelopers.com/magazine/216-the-prompt-engineer)

## Related Jobs

- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Staff Developer Advocate, GitHub Security Lab](https://www.wearedevelopers.com/jobs/ext/1921051-staff-developer-advocate-github-security-lab) at **GitHub**
- [Engineer, Offensive Security Organization](https://www.wearedevelopers.com/jobs/ext/1992296-engineer-offensive-security-organization) at **Twilio**
- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/588393-machine-learning-engineer) at **Twilio**
- [AI & Machine Learning Engineer (all genders)](https://www.wearedevelopers.com/jobs/48217-ai-machine-learning-engineer-all-genders) at **msg**