Definition
Plain language
When output is so malformed that the software reading it can't make sense of it at all.
As stated in the literature
A failure to extract structured content from a model's generation because it violates the expected format; distinct from a semantically wrong but well-formed answer.
Also called: parse errors
Why it matters: Parse errors can silently depress benchmark scores or break automated pipelines for reasons that have nothing to do with whether the model actually knew the answer.
For example, a program expecting the answer inside curly braces gets a chatty paragraph with no braces at all, so it throws the response away without ever judging whether the answer was right.
Heard on the show
“And it gets worse as you push harder — parse errors climb from twenty-nine out of seventy-two up to sixty out of seventy-two.”Episode 237 — The Model Built a Perfect Map of the Puzzle, Then Lost It