Definition
Prompt injection is an attack where adversarial instructions are smuggled into data that a model later reads — a web page, an email, a tool output — causing the model to ignore its real instructions and follow the injected ones. It’s the defining security problem of LLM agents.
Episodes covering this
Worth reading next
Papers we haven't done a deep dive on yet, but would recommend on this topic.
- InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
- Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
- CaMeL: How to make LLM agents safe
- Injecting Relevance: Exploring the Role of Prompt Injection Attacks on Search Engine Optimization
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
- LlamaFirewall: An Open Source Guardrail System for Building Secure AI Agents
- The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
- Ignore Previous Prompt: Attack Techniques For Language Models
- Multimodal Neurons in Artificial Neural Networks
- Universal and Transferable Adversarial Attacks on Aligned Language Models