Definition
Plain language
Being able to send something questions and see its answers, without seeing anything about how it works inside.
As stated in the literature
An attacker or evaluator model in which only sampled outputs are observable — no weights, activations, or token probabilities. Contrasts with white-box access.
Also called: black-box, black box access
Why it matters: It defines the realistic threat model for most deployed AI systems, since anyone with an ordinary account can mount attacks that need nothing more than questions and answers.
For example, someone using a chatbot through a public website can type prompts and read replies, but has no way to inspect the model's internal numbers.
Heard on the show
“There's one recent image-only line in visual-document retrieval, but its main attack leans on gradient-based optimization, and that degrades badly under black-box transfer.”Episode 247 — One Edited Photo, an Honest Caption, and a RAG System That Believes It