Definition
Plain language
The finding that an AI's decision to refuse or comply is set in the first few words, not built up through careful thinking.
As stated in the literature
The result that safety alignment in instruction-tuned models is concentrated in the opening tokens of a response (or, for reasoning models, during prompt reading), explaining the effectiveness of prefix-manipulation jailbreaks.
Also called: shallow safety alignment
Why it matters: It explains why tricks that force a particular opening to a response can bypass safety training, exposing a real weakness in how alignment is built.
For example, a model's choice to refuse a harmful request is often locked in by its first few words, not reached after careful deliberation.
Heard on the show
“… every technical term tap-to-define, with links to the related papers grouped by theme, from shallow alignment to chain-of-thought faithfulness. …”Episode 171 — The Safety Decision a Model Makes Before It Thinks a Word