Glossary · Term

harness

← all terms

Definition

Plain language

The wrapper of code, tools, and prompts around an AI agent that shapes how it actually behaves.

As stated in the literature

The surrounding software stack for an LLM agent — tool definitions, parsers, system prompts, scratchpad conventions, and orchestration logic — that determines in-context behavior beyond raw model weights.

Also called: agent harness, harnesses

Why it matters: Much of what looks like model capability is actually harness engineering, which is why fair agent comparisons require fixing the harness.

For example, two papers might use the same base model but report very different agent scores because their harnesses parse tool calls and format prompts differently.

Heard on the show

“So they built a real harness — an actual client and server talking to each other — and for each of the eight attacks they recorded four yes-or-no facts.”
Episode 208 — The Blank Space in Your AI Approval Box That Isn't Empty

Mentioned in 48 episodes

  1. 208
    The Blank Space in Your AI Approval Box That Isn't Empty
  2. 196
    AI Agents Reached Opposite Conclusions From the Same Data — and Passed Review
  3. 195
    Why 'Be Careful' Does Nothing for AI Coding Agents, and What Does
  4. 190
    The Skill Every AI Manager Is Missing: Handing Out Exactly the Right Keys
  5. 189
    Why Phone Agents Ace the Test and Crash on Your Actual Phone
  6. 180
    The Bug Where Smart Assistants Read a Fact and Still Forget It
  7. 175
    One Crosscoder Feature Flips a Stalling Chatbot Into a Working Agent
  8. 174
    When the AI 'Schemes,' It's Usually Just Lazy or Confused
  9. 171
    The Safety Decision a Model Makes Before It Thinks a Word
  10. 168
    When Turning Experience Into Code Makes Your AI Agent Dumber
  11. 166
    A Router That Beats the Frontier Models It Calls
  12. 164
    The Summarizer That Quietly Deletes Your Agent's Safety Rules
  13. 163
    Why Training Only on Perfect Solutions Cripples a Model's Reasoning
  14. 159
    Can a Coding Agent Run Its Own Robot Experiments Overnight, With No Human Resetting the Scene?
  15. 157
    When an AI Coding Agent Drives a Phone Through the Terminal, No Screen Needed
  16. 147
    Agents Fail at the Body, Not the Brain: A Self-Rewriting Scaffold That Lifts a 9B Model 44 Points
  17. 142
    Training a Tiny Model to Run the Plumbing Between an Agent and the World
  18. 139
    When Optimizing One GPU Kernel Quietly Breaks the Whole System
  19. 131
    Why Autonomous Research Agents Forget Their Own Lessons, and Arbor's Fix
  20. 129
    How a Crowd of Anonymous AI Agents Broke a 40-Year Math Record
  21. 126
    How Coding Agents Can Mine Their Own Failures Into a Self-Targeting Curriculum
  22. 125
    AI Coding Agents Run a Marathon, and Fewer Than One in Three Finish
  23. 124
    A Cheap Model With the Blueprints Beats Expensive Models Working Blind
  24. 121
    When the Agent Says It's Done But Nothing Happened: Debugging the Harness, Not the Model
  25. 120
    How an AI Agent Rewrites Its Own Tools, Without an Answer Key
  26. 113
    What If a Prompt Injection Never Left? Attacks That Wait in Agent Memory
  27. 111
    How a 4B Web Agent Beat Models 60x Its Size on 500 Demonstrations
  28. 109
    An AI Got Caught Reading the Answer Key, And Why That Catch Matters
  29. 105
    The Trojan Is Your Agent's Memory: Why Single-Step Defenses Miss Persistent Attacks
  30. 101
    Treating Math Formalization Like a Codebase, and Where the Agents Cheat
  31. 097
    Same Tokens, Same Cost, Wildly Different Results: What Actually Scales in AI Agents
  32. 088
    Two Levers for Self-Improving AI: When Rewriting Code Isn't Enough
  33. 086
    Why Frozen-Weight Agents Still Get Worse Over Time
  34. 084
    Terminal Agents Get Free Supervision From The Tokens We've Been Throwing Away
  35. 078
    Training a Markdown File: When LLM Self-Improvement Borrows the Discipline of Neural Net Training
  36. 071
    When the Model Is Fine and the Plumbing Is Broken: Fixing Agents at the Interface
  37. 067
    An AI Just Solved a 1996 Erdős Problem—and the Simplest Agent Won
  38. 061
    When Helpful Agents Go Sideways: A 404 Error, Campus Security, and Why Alignment Misses This
  39. 053
    An AI Agent Swapped In Focal Loss And Beat A Human-Tuned Training Script
  40. 047
    When Agent Benchmarks Lie: The Harness Problem in Open-Source AI
  41. 046
    When the AI Optimizer Edits the Grade Book: Why Harnessing Evolution Needs a Wall
  42. 045
    When a Frontier Model Talks Its Own Twin Into Climate Denial
  43. 029
    Why Forty-Eight Percent on FrontierMath Isn't the Real Story in DeepMind's New Math Paper
  44. 028
    Teaching a Model to Hire Copies of Itself: Recursive Agent Optimization
  45. 024
    An AI Agent That Found 28 Zero-Days in Windows — And What Made It Work
  46. 014
    Why a Constrained Pipeline Beat a Full Coding Agent at Finding Bugs 30-to-1
  47. 007
    Exploration Hacking: When Models Sabotage Their Own RL Training
  48. 001
    When AI Models Quietly Protect Each Other From Shutdown

Related terms

agent scratchpad system prompt weights