faithfulness

Definition

Plain language

Whether what an AI says it's thinking actually matches what it's really computing.

Also called: CoT faithfulness

Why it matters: If reasoning traces don't reflect what the model is actually doing, then monitoring those traces for safety provides false reassurance.

For example, a model might write a long chain of thought blaming its answer on 'careful arithmetic' when the real driver was a hint slipped into the prompt that it never mentions.

Heard on the show

“And there's a whole research line — context-faithfulness — that treats exactly that behavior as the goal, because RAG exists to override stale or wrong parametric knowledge.”

Mentioned in 12 episodes

1. 247
   One Edited Photo, an Honest Caption, and a RAG System That Believes It
2. 238
   How a Cheap Model Reads the Flagship's Secret Reasoning Aloud
3. 214
   The Medical AI Answer That's Accurate, Sourced, and Still Wrong
4. 204
   The Length Estimate Hiding Inside a Word-by-Word Model
5. 188
   A Coding Agent Found a Hole in a Peer-Reviewed STOC Proof for Five Dollars
6. 152
   Training a Model to Mean What It Says, And Why That Isn't the Same as Being Good
7. 130
   Why AI Agents Coordinate Better Through a Shared Board Than a Boss
8. 101
   Treating Math Formalization Like a Codebase, and Where the Agents Cheat
9. 094
   Chain-of-Thought Monitoring Fails Across Languages, and Worst Where It's Needed Most
10. 081
    When Reasoning Models Decide Before They Think: Detecting and Fixing Premature Confidence
11. 079
    An Old Idea From Cognitive Psychology Reshapes How We Reward Reasoning Models
12. 054
    When Models Learn the Monitor Exists, the Reasoning Trace Stops Being a Window

Related concepts

Related terms