LLM-as-a-judge

Definition

Using one AI model to grade what another AI model produced.

An evaluation or moderation pattern in which a language model serves as the grader of outputs, preferences, or safety properties.

Also called: LLM judge, LLM-as-judge, LLM-as-a-Judge

Mentioned in 7 episodes

1. 062
   Treating Hallucinations as Exploits: A Gate-Based Architecture for Agent Safety
2. 059
   Firefly's Inversion: Building Verified Tool-Call Training Data by Working Backward
3. 055
   Why LLM Judges Flip Their Verdicts When You Change the Question Format
4. 052
   An Old Reinforcement Learning Tradeoff Sneaks Back Into LLM Agents
5. 028
   Teaching a Model to Hire Copies of Itself: Recursive Agent Optimization
6. 023
   Why a Small Agent Confidently Overwrites Memories It Doesn't Understand
7. 020
   The Compliance Gap: Why AI Says Yes and Does No

Related concepts