Glossary · Term

harness

← all terms

Definition

Plain language

The wrapper of code, tools, and prompts around an AI agent that shapes how it actually behaves.

As stated in the literature

The surrounding software stack for an LLM agent — tool definitions, parsers, system prompts, scratchpad conventions, and orchestration logic — that determines in-context behavior beyond raw model weights.

Also called: agent harness, harnesses

Why it matters: Much of what looks like model capability is actually harness engineering, which is why fair agent comparisons require fixing the harness.

For example, two papers might use the same base model but report very different agent scores because their harnesses parse tool calls and format prompts differently.

Heard on the show

“That feels like you're testing the harness's amnesia rather than the model.”
Episode 245 — Fifteen Models Ran Football Clubs for Twenty Years, and Size Didn't Decide It

Mentioned in 50 episodes

  1. 245
    Fifteen Models Ran Football Clubs for Twenty Years, and Size Didn't Decide It
  2. 240
    Frontier Models Designed Follow-Ups To Fraudulent Papers 93% Of The Time
  3. 208
    The Blank Space in Your AI Approval Box That Isn't Empty
  4. 196
    AI Agents Reached Opposite Conclusions From the Same Data — and Passed Review
  5. 195
    Why 'Be Careful' Does Nothing for AI Coding Agents, and What Does
  6. 190
    The Skill Every AI Manager Is Missing: Handing Out Exactly the Right Keys
  7. 189
    Why Phone Agents Ace the Test and Crash on Your Actual Phone
  8. 180
    The Bug Where Smart Assistants Read a Fact and Still Forget It
  9. 175
    One Crosscoder Feature Flips a Stalling Chatbot Into a Working Agent
  10. 174
    When the AI 'Schemes,' It's Usually Just Lazy or Confused
  11. 171
    The Safety Decision a Model Makes Before It Thinks a Word
  12. 168
    When Turning Experience Into Code Makes Your AI Agent Dumber
  13. 166
    A Router That Beats the Frontier Models It Calls
  14. 164
    The Summarizer That Quietly Deletes Your Agent's Safety Rules
  15. 163
    Why Training Only on Perfect Solutions Cripples a Model's Reasoning
  16. 159
    Can a Coding Agent Run Its Own Robot Experiments Overnight, With No Human Resetting the Scene?
  17. 157
    When an AI Coding Agent Drives a Phone Through the Terminal, No Screen Needed
  18. 147
    Agents Fail at the Body, Not the Brain: A Self-Rewriting Scaffold That Lifts a 9B Model 44 Points
  19. 142
    Training a Tiny Model to Run the Plumbing Between an Agent and the World
  20. 139
    When Optimizing One GPU Kernel Quietly Breaks the Whole System
  21. 131
    Why Autonomous Research Agents Forget Their Own Lessons, and Arbor's Fix
  22. 129
    How a Crowd of Anonymous AI Agents Broke a 40-Year Math Record
  23. 126
    How Coding Agents Can Mine Their Own Failures Into a Self-Targeting Curriculum
  24. 125
    AI Coding Agents Run a Marathon, and Fewer Than One in Three Finish
  25. 124
    A Cheap Model With the Blueprints Beats Expensive Models Working Blind
  26. 121
    When the Agent Says It's Done But Nothing Happened: Debugging the Harness, Not the Model
  27. 120
    How an AI Agent Rewrites Its Own Tools, Without an Answer Key
  28. 113
    What If a Prompt Injection Never Left? Attacks That Wait in Agent Memory
  29. 111
    How a 4B Web Agent Beat Models 60x Its Size on 500 Demonstrations
  30. 109
    An AI Got Caught Reading the Answer Key, And Why That Catch Matters
  31. 105
    The Trojan Is Your Agent's Memory: Why Single-Step Defenses Miss Persistent Attacks
  32. 101
    Treating Math Formalization Like a Codebase, and Where the Agents Cheat
  33. 097
    Same Tokens, Same Cost, Wildly Different Results: What Actually Scales in AI Agents
  34. 088
    Two Levers for Self-Improving AI: When Rewriting Code Isn't Enough
  35. 086
    Why Frozen-Weight Agents Still Get Worse Over Time
  36. 084
    Terminal Agents Get Free Supervision From The Tokens We've Been Throwing Away
  37. 078
    Training a Markdown File: When LLM Self-Improvement Borrows the Discipline of Neural Net Training
  38. 071
    When the Model Is Fine and the Plumbing Is Broken: Fixing Agents at the Interface
  39. 067
    An AI Just Solved a 1996 Erdős Problem—and the Simplest Agent Won
  40. 061
    When Helpful Agents Go Sideways: A 404 Error, Campus Security, and Why Alignment Misses This
  41. 053
    An AI Agent Swapped In Focal Loss And Beat A Human-Tuned Training Script
  42. 047
    When Agent Benchmarks Lie: The Harness Problem in Open-Source AI
  43. 046
    When the AI Optimizer Edits the Grade Book: Why Harnessing Evolution Needs a Wall
  44. 045
    When a Frontier Model Talks Its Own Twin Into Climate Denial
  45. 029
    Why Forty-Eight Percent on FrontierMath Isn't the Real Story in DeepMind's New Math Paper
  46. 028
    Teaching a Model to Hire Copies of Itself: Recursive Agent Optimization
  47. 024
    An AI Agent That Found 28 Zero-Days in Windows — And What Made It Work
  48. 014
    Why a Constrained Pipeline Beat a Full Coding Agent at Finding Bugs 30-to-1
  49. 007
    Exploration Hacking: When Models Sabotage Their Own RL Training
  50. 001
    When AI Models Quietly Protect Each Other From Shutdown

Related terms

agent scratchpad system prompt weights