Glossary · Term

GitHub

← all terms

Definition

Plain language

The dominant online home for code projects and where most open-source software lives.

As stated in the literature

The widely used hosted version-control and collaboration platform for software development; many agent benchmarks draw from real GitHub issues and pull requests.

Why it matters: It's where most modern software gets built, so it's also the natural source of realistic tasks for evaluating coding agents.

For example, the SWE-Bench benchmark draws its tasks directly from real bug reports and pull requests on public GitHub repositories.

Heard on the show

“On SWE-Bench Lite — real GitHub-issue patches — with Claude Haiku, the Bayesian controllers beat "always verify" by roughly sixty-two to sixty-eight utility units per instance.”
Episode 170 — When a One-Liner Beats Your Agent's Clever Verification Logic

Mentioned in 21 episodes

  1. 170
    When a One-Liner Beats Your Agent's Clever Verification Logic
  2. 169
    Why Better Bug Reports Can Make AI Coding Agents Worse
  3. 146
    How an Innocent README Can Freeze an AI Agent's Safety Check for an Hour
  4. 142
    Training a Tiny Model to Run the Plumbing Between an Agent and the World
  5. 130
    Why AI Agents Coordinate Better Through a Shared Board Than a Boss
  6. 129
    How a Crowd of Anonymous AI Agents Broke a 40-Year Math Record
  7. 126
    How Coding Agents Can Mine Their Own Failures Into a Self-Targeting Curriculum
  8. 125
    AI Coding Agents Run a Marathon, and Fewer Than One in Three Finish
  9. 122
    When Your Coding Agent Lies About the Fix: Verifying the Plan Before the Model Runs
  10. 112
    When an AI Agent Cheats Without Being Told: Inside the Meta-Agent Challenge
  11. 105
    The Trojan Is Your Agent's Memory: Why Single-Step Defenses Miss Persistent Attacks
  12. 093
    A Calibrated Knob for Weak-to-Strong AI Oversight, Tested on Real Code
  13. 090
    How MiniMax-M2 Bets That Sparsity Plus Verifiable Rewards Can Match Frontier Agents
  14. 072
    A Robot Made Graphene Without Help, And Caught Itself Hallucinating
  15. 068
    The OS Trick That Makes Tree Search Practical for Coding Agents
  16. 061
    When Helpful Agents Go Sideways: A 404 Error, Campus Security, and Why Alignment Misses This
  17. 058
    Why Upgrading Your AI Auditor to a Smarter Model Can Make Your System Less Safe
  18. 047
    When Agent Benchmarks Lie: The Harness Problem in Open-Source AI
  19. 012
    Why AI Coding Agents Keep Trying to Debug Without a Debugger
  20. 005
    Why a Debugger Designed for Humans Is the Wrong Tool for an AI Agent
  21. 003
    How to Pick the Best of Sixteen Coding Agent Rollouts

Related concepts

CodeQL SWE-bench

Related terms

agent