Definition
AI security covers two related concerns: protecting AI systems from attacks (prompt injection, weight exfiltration, adversarial inputs) and the use of AI itself in offensive and defensive cyber operations. The two are tangled because the same model that helps you audit your code can help an attacker write an exploit.
Episodes covering this
Worth reading next
Papers we haven't done a deep dive on yet, but would recommend on this topic.
- AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents
- Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
- CaMeL: How to make LLM agents safe
- Injecting Relevance: Exploring the Role of Prompt Injection Attacks on Search Engine Optimization
- BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
- Bypassing Backdoor Detection Algorithms in Deep Learning
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
- LlamaFirewall: An Open Source Guardrail System for Building Secure AI Agents
- The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
- Ignore Previous Prompt: Attack Techniques For Language Models
- Universal and Transferable Adversarial Attacks on Aligned Language Models
- Many-shot Jailbreaking
- Tamper-Resistant Safeguards for Open-Weight LLMs
- TruFor: Leveraging All-Round Clues for Trustworthy Image Forgery Detection and Localization