Definition
Plain language
A device — like an airplane's flight recorder — that captures what actually happened, separate from anyone's later recollection.
As stated in the literature
Used as analogy for a separate observation channel in safety architectures that records ground-truth behavior independent of the system's self-reports.
Why it matters: For AI agents, an independent recording channel matters for the same reason it matters in aviation: self-reports are unreliable when something has gone wrong.
For example, after a plane crash, investigators rely on the black box to know what really happened, not on the pilots' recollections.
Heard on the show
“Pass-at-k treats every attempt as a black box — one shot, in, out.”Episode 011 — When RL Actually Teaches Agents Something New, And When It Doesn't