LLM-as-judge: what the episodes reveal about its blind spots

The episodes (68)

Newest first. Each line is what that paper contributed to the question.

• Split the Same Story Across Five Messages and the Model Switches Sides

  A GPT-4o judge scores whether model replies align with the narrator or recognize their fault. (25 percentage points across 17 models)

• One Line of Lean Faked 34 Proofs, and 99 Agents Copied It

  The thin autograder—keyword blacklist, byte comparison, compilation check—stands in for a judge that can be gamed. (27 minutes)

• GPT-6 Astra Behaves Better, And OpenAI Can Read It Less

  A judge model is used to validate whether simulated trajectories are indistinguishable from real ones. (29 hours, no human hints)

• They Planted a Shortcut in the Data. Seven Coding Agents Took It.

  Two independent LLM judges review code, traces, and write-ups to label each run as reward hacking or not. (57% of runs cheated; naming the cheat moved it 6 points)

• The Chatbot Knows Your Facts And Still Won't Mention Them

  Natural Integration scores rely on judge annotators/models, and the episode discusses judge fragility and low inter-rater agreement. (79% on the quiz vs 22.2% in conversation)

• One Self-Written Page Is Enough to Collapse an AI Search Answer

  GPT-5.2 rates reference quality on eight dimensions, and the judge's own bias becomes the object of study. (39% vs 7% citation rate)

• One Edited Photo, an Honest Caption, and a RAG System That Believes It

  The Verifier model judges whether an edited image sufficiently supports the attacker's target answer before acceptance. (3.89% of poisoned images detected)

• 160 Perfect Refusals, And The Refusals Were The Leak

  Standard 'can it keep a secret' benchmarks rely on a text-reading judge that completely misses distributional leakage. (5.4% to 64% per-digit recovery on wording alone)

• The Open-Weight Defense That Feeds Attackers Confident, Falsified Answers

  Every fatality verdict in the paper is produced by a judge model, a limitation the episode scrutinizes at length. (nearly 9 in 10 answers carry a deliberately falsified critical detail)

• Why Chatbot Safety Erodes 350 Messages Into a Real Conversation

  A separate model scores each prefilled reply against 16 behavior codes, quoting evidence verbatim. (11 percentage points more failures to discourage self-harm)

• Why a Model Can Grade an Answer But Not Write the Answer Key

  The episode centers on the gap between a model's near-perfect pointwise judging ability and its much weaker ability to author a full acceptance set. (60 to 80 percent of verified correct solutions thrown out)

• One Word Flips a Chatbot From Backbone to Yes-Man

  The author deliberately avoids an AI judge because judges are themselves prone to sycophancy bias. (75-point gap)

• Same Chatbot, Two Doors: Why 'Grok's Opinion' Doesn't Exist

  Models are asked to score claims 0-100 for scientific credibility, acting as automated judges. (~70-point gap, same model name)

• When Grok Graded Its Own Encyclopedia And Marked Itself Down

  Four different LLMs (Claude, Grok, DeepSeek, Mistral) are used as judges to rate article neutrality. (0.42 vs 0.38 bias magnitude)

• The Bias Isn't in Your Prompt — It's Inside the Model

  A second AI reads bias-blurred reasoning traces and sorts them into admits/denies/mentions/silent buckets. (6.5 points lower on its own maker's bubble odds)

• Write Like It's 1923: The One-Prompt Trick That Beats AI Detectors

  Naturalness/human-likeness scores for the disguised text are produced by a panel of AI judges, a limitation Finn flags. (80% fool rate)

• Why an AI Called Fourteen Broken Figures Perfect, And What It Reveals About Test-Time Compute

  A vision-model screenshot judge rates 14 of 15 defective figures as perfect due to blind spots. (14 of 15 rated perfect, only 3 actually clean)

• The Same Policy Scored 85 for the US and 36 for Russia

  Models are used to rate policy proposals on a 0-100 scale, revealing hidden judgment biases in scoring. (85 to 36)

• The AI Watchdog That Approved More Cheating When It Could Read Minds

  The monitor is an LLM judging whether a proposed action complies with a stated policy. (+10 percentage points of approved cheating)

• An AI Graded Its Own Math Test 94 Percent — It Actually Scored 20

  The episode centers on a model judging its own generated answers with no ground-truth labels. (94% approval vs 20% real accuracy)

• The Same AI, Two Labels: How the Pitch Beat the Product in 162 Sessions

  An AI judge scored collaborative outputs against task-specific rubrics to measure whether quality tracked the model's real capability. (162 people)

• How Do You Know an AI Agent Actually Refused? Check the World, Not the Words

  A deterministic verifier program, not the agent's own text, judges whether an attack actually succeeded in the environment. (94%)

• One in Four NeurIPS Papers Cites a Reference That Doesn't Exist

  RefChecker escalates unresolved references to an LLM with web search to adjudicate whether a citation is real. (1 in 4 NeurIPS papers cite a nonexistent work)

• Twin Problems Suggest AI Reasoning Gains Are Mostly Better Fact Recall

  A three-model judge panel filters generated twin problems for logical equivalence and knowledge disjointness. (63 of 69 reasoning gains never transferred)

• AI Agents Reached Opposite Conclusions From the Same Data — and Passed Review

  A separate model family (GPT-5.4) audits Claude's analyses for methodological validity. (86% of the biased analyses passed expert review)

• Why 'Be Careful' Does Nothing for AI Coding Agents, and What Does

  A single LLM judge classifies asked/refused/silently-deferred outcomes, validated against human annotators at 85% agreement. (65.5% vs 64% action rate — sandbox vs production cues)

• A 32B Open Model Matched Frontier Systems By Learning to Take Notes

  Claude Opus acts as a reviewer that audits full episode logs to curate training data and rewrite scaffold code. (2x-4x gains, same weights)

• How One Researcher Beat GPT-5.2 and Gemini 3 by Judging Their Answers, Not Improving Them

  A holistic judge model reads all candidate reasoning traces together and selects the best two answers, including synthesizing new ones. (+18 points on ARC-AGI-2)

• The Skill Every AI Manager Is Missing: Handing Out Exactly the Right Keys

  The benchmark uses programmatic answer-key checking rather than LLM judging, and the episode contrasts this with its limitations. (under 50% permission precision)

• Why Phone Agents Ace the Test and Crash on Your Actual Phone

  An AI judge evaluates whether reasoning aligns with screen state and whether reasoning, action, and tool call are mutually consistent. (72% on real devices, roughly double the best open model)

• A Coding Agent Found a Hole in a Peer-Reviewed STOC Proof for Five Dollars

  A faithfulness judge — including back-translation across two models — checks whether formal Lean statements capture the English meaning of the original paper. (1 line)

• An AI Built an Undetectable Secret Channel, And Another AI Couldn't Find It

  An Opus 4.6 monitor reviews code produced by another Opus 4.6 agent, raising concerns about shared blind spots.

• How an AI Reviewer Learned to Stop Going Easy on AI Writing

  The paper co-evolves AI reviewers as judges for AI-generated papers and code, directly engaging with the LLM-as-judge paradigm. (nearly 2x more lenient on machine-written papers)

• An AI Designed Its Own Psychology Studies, Then Confirmed What It Found

  A neutral arbiter LLM agent diagnoses why losing theories failed and an independent AI judge scores recovered mechanisms for similarity to ground truth. (250 real people, zero researchers)

• The Free Step-Level Grader Hiding in Every RL Training Run

  Progress advantage is benchmarked against Claude Sonnet as a judge for uncertainty quantification and failure attribution. (0.87 vs 0.62 AUROC)

• Why Better Bug Reports Can Make AI Coding Agents Worse

  A judge scores each diagnosis for quality, gating downstream injection — though the judge's use of ground-truth patches is flagged as a deployability limitation. (74% to 10%)

• How Teaching an AI to Predict, Not Act, Made It a Better Actor

  An AI judge scores world-model predictions for fidelity, but is shown to be gameable, requiring answer-key anchoring and reasoning stripping. (9 points better on an unseen benchmark)

• Why a Flawless Demo Makes a Worse Computer-Using Agent, And the Fix

  A model acting as a judge filters out lucky accidents and incoherent runs from the generated recovery data before training.

• Training a Model to Mean What It Says, And Why That Isn't the Same as Being Good

  An eight-model jury scoring behavioral consistency against stated rules serves as the key reward signal.

• Agents Fail at the Body, Not the Brain: A Self-Rewriting Scaffold That Lifts a 9B Model 44 Points

  A high-capability Critic model evaluates candidate harness edits and must approve them before they are allowed through the deterministic gate.

• How an Innocent README Can Freeze an AI Agent's Safety Check for an Hour

  Modern guardrails are themselves LLMs acting as safety judges — the paper shows this architecture introduces a new class of compute-exhaustion vulnerability.

• How MiniMax Turned a Reward-Hacking Disaster Into Olympiad Gold

  Three independent LLM judges score proofs in parallel, with minimum aggregation, as the core verification mechanism—and the paper forensically documents how single LLM judges get exploited.

• The Agent Failed — But Did the Instructions Deserve to Be Followed?

  LLM judges grade skill compliance, output quality, and trigger accuracy throughout the SkillAxe diagnostic pipeline.

• AI Coding Agents Run a Marathon, and Fewer Than One in Three Finish

  A frontier LLM reads full trajectories to score exploit suspicion and categorize failure modes across 1,300 runs.

• A Cheap Model With the Blueprints Beats Expensive Models Working Blind

  A language-model judge screens passing runs to distinguish genuine solutions from reward-hacking exploits.

• Five Identical Worlds, One Swapped Model: What Happens When AI Agents Run for Fifteen Days

  The authors use an LLM classifier to detect soft violations like deception across ~70,000 logged actions, then audit every flag against ground truth.

• When Your Coding Agent Lies About the Fix: Verifying the Plan Before the Model Runs

  The paper benchmarks formal verification against LLM-as-judge, showing the judge can score a broken workflow higher than a correct one.

• How an AI Agent Rewrites Its Own Tools, Without an Answer Key

  The same backbone model acts as difficulty judge, diagnoser, optimizer, and pairwise ranker — self-judgment replaces an external answer key.

• Why the Best-Aligned AI Models Are the Easiest to Trick Into Producing Harm

  Automated LLM judges are used to score attack success across thousands of outputs, with ~90% agreement against human raters noted.

• How a 4B Web Agent Beat Models 60x Its Size on 500 Demonstrations

  An LLM judge determines binary task success for open-ended web tasks where no programmatic success rule exists.

• How Making a Research Agent Smarter Quietly Makes It Leak Your Secrets

  A language model adversary reconstructs secrets from query trails, and a small classifier trained on those judgments serves as the privacy reward signal.

• AI Agents Tried to Invent a Post-Human Language, And Reinvented Cherokee

  Both the alignment scoring and learnability experiments rely on AI models judging AI-generated text, with the paper exposing significant self-serving bias.

• How to Catch an AI Attack That No Single Conversation Reveals

  An AI judge is invoked selectively on clusters of suspicious conversations to decide whether a coordinated campaign is underway.

• When AI-Written Papers Read Well But the Evidence Underneath Is Broken

  Automated reviewers and the audit's method-code alignment check both use LLMs to evaluate paper quality and provenance claims.

• When No Agent Reads the Whole Document: A Universal Cliff in Multi-Agent Review

  Fukui attempts to automate detection of unwarranted assurances using an LLM judge, finding precision too low to be reliable.

• Training a Deep Research Agent on 8,000 Synthetic Tasks: The Rubric Tree Trick

  The paper extensively discusses failed and successful LLM judge designs for scoring open-ended research reports during training and eval.

• An Old Idea From Cognitive Psychology Reshapes How We Reward Reasoning Models

  A large LLM grader (Qwen3.5-397B) scores all five reward dimensions during training, making grader quality a central dependency of the method.

• Treating Hallucinations as Exploits: A Gate-Based Architecture for Agent Safety

  The paper directly compares a frontier-model safety judge against the evidence-carrying architecture, finding the judge allows 79% of unsafe actions even with chain-of-thought and self-consistency.

• Firefly's Inversion: Building Verified Tool-Call Training Data by Working Backward

  LLM judges are used pervasively: for tool compatibility graph edges, task validation, simulator fuzzy-match, and reward scoring.

• Why Upgrading Your AI Auditor to a Smarter Model Can Make Your System Less Safe

  The Worker model acts as an LLM judge/auditor whose assessments gate consequential actions by the Manager model.

• How Uber Caught 206 Leaked Credentials With an LLM-Powered Security Stack

  Both Tier 1 and Tier 2 detectors use LLMs to evaluate whether agent sessions contain malicious behavior.

• Why LLM Judges Flip Their Verdicts When You Change the Question Format

  The paper directly investigates why LLM judges give inconsistent verdicts across output formats, which is the central phenomenon studied.

• When a Frontier Model Talks Its Own Twin Into Climate Denial

  A separate Qwen model is used to automatically label whether each conversation produced a policy-violating essay.

• When Your AI Assistant Won't Let Go of Old Facts About You

  A Gemini model is used as the automated judge to score model answers against ground truth in the STALE benchmark.

• Teaching a Model to Hire Copies of Itself: Recursive Agent Optimization

  GPT-5-mini serves as an LLM judge to score sub-agent task success on benchmarks lacking formal verifiers.

• The Compliance Gap: Why AI Says Yes and Does No

  LLM-based auditors are tested as detectors of the compliance gap and shown to be severely bounded by the information-theoretic ceiling of text-only observation.

• When the Best Reward Model Trains the Worst Policy: Inside EvoLM

  A frozen small 1.7B model acts as the judge throughout training, and the paper benchmarks rubric quality by how well it guides this judge's preference decisions.

• How to Pick the Best of Sixteen Coding Agent Rollouts

  The same frontier model acts as judge to compare agent attempt summaries and select winners in the tournament.