Glossary · Term

inference

← all terms

Definition

Plain language

Running a finished AI model to get an answer, as opposed to training it in the first place.

As stated in the literature

The forward-pass execution of a trained model to produce outputs, distinct from training; the regime where serving cost, latency, KV-cache memory, and test-time scaling techniques live.

Also called: inference-time, inference time

Why it matters: It is where the real-world costs of running AI live, since speed, memory use, and serving expense all determine whether a model is practical to deploy.

For example, every time you type a question into a chatbot and get an answer, that's inference, separate from the earlier training that built the model.

Heard on the show

“… footnotes, one sentence apiece: the whole thing costs about three-point-six times a single agent's inference, though it beats five-way voting while spending less than voting's token budget; and shuffling …”
Episode 200 — The One Mechanism That Turns Twenty AI Clones Into an Actual Team

Mentioned in 72 episodes

  1. 200
    The One Mechanism That Turns Twenty AI Clones Into an Actual Team
  2. 198
    The Model That Knows the Answer and Can't Say It
  3. 197
    Twin Problems Suggest AI Reasoning Gains Are Mostly Better Fact Recall
  4. 191
    How One Researcher Beat GPT-5.2 and Gemini 3 by Judging Their Answers, Not Improving Them
  5. 185
    Aligned to Refuse, Built to Tap: When Phone Agents Know the Task Is a Crime and Do It Anyway
  6. 183
    Why You Can't Fine-Tune Foresight Into an AI Agent
  7. 179
    How DeepSeek Made One User Faster Without Slowing Down the Crowd
  8. 177
    Why Raw Profiler Data Made an AI Worse at Writing GPU Code
  9. 175
    One Crosscoder Feature Flips a Stalling Chatbot Into a Working Agent
  10. 174
    When the AI 'Schemes,' It's Usually Just Lazy or Confused
  11. 171
    The Safety Decision a Model Makes Before It Thinks a Word
  12. 165
    A Free-Lunch Tweak That Lets a Tiny Agent Beat Frontier Giants
  13. 158
    How Floating-Point Rounding Lets a Model Tell Which Chip It's On — And Misbehave
  14. 156
    Why More Human Demonstrations Made a Computer-Use Agent Worse
  15. 151
    Why More Experience Made This AI Agent Worse, And How to Fix It
  16. 150
    Don't Kill the Loser: A Different Way to Handle Two AI Agents Colliding
  17. 149
    When Cornering a Chatbot Makes It Lie: J.P. Morgan's Case for 'Playing Dead'
  18. 146
    How an Innocent README Can Freeze an AI Agent's Safety Check for an Hour
  19. 141
    How Two Tokens Reopened a Reasoning Method the Field Had Given Up On
  20. 140
    When a Reasoning Model Says "Let Me Double-Check" After It's Already Decided
  21. 139
    When Optimizing One GPU Kernel Quietly Breaks the Whole System
  22. 133
    How MiniMax Turned a Reward-Hacking Disaster Into Olympiad Gold
  23. 127
    What Diffusion Language Models Were Missing: A Map, Not an Algorithm
  24. 119
    Beating Reinforcement Learning Without Ever Touching the Model's Weights
  25. 116
    Why Streaming Half a Reasoning Chain Beats Sending the Whole Thing
  26. 115
    Teaching a Phone Agent to Reason Silently, And Keeping It Honest
  27. 109
    An AI Got Caught Reading the Answer Key, And Why That Catch Matters
  28. 108
    The Reasoning Cliff: Why Thinking Longer Makes Models Worse at Exact Step-by-Step Tasks
  29. 107
    How a Market of Crippled AI Agents Outscored One Unrestricted Model
  30. 106
    Giving Agents a Notebook Instead of New Weights: How ExpGraph Lets Frozen Models Learn
  31. 100
    How a Prompt Wrapper Lets a Frontier Model Play Poker Like an Expert
  32. 099
    How an Open-Book Trick Teaches a Model to Catch Its Own Mistakes
  33. 098
    Finding Millions of Readable Concepts Inside a Real, Deployed AI Model
  34. 097
    Same Tokens, Same Cost, Wildly Different Results: What Actually Scales in AI Agents
  35. 092
    When Search Agents Don't Really Search: The Memory Shortcut Hiding in Browsing Benchmarks
  36. 091
    When Better Fine-Tuning Can't Help: A Geometric Impossibility in LLM Causal Reasoning
  37. 090
    How MiniMax-M2 Bets That Sparsity Plus Verifiable Rewards Can Match Frontier Agents
  38. 086
    Why Frozen-Weight Agents Still Get Worse Over Time
  39. 085
    Why Long-Context Models Might Need Compute, Not Capacity, Before Eviction
  40. 082
    Training a Deep Research Agent on 8,000 Synthetic Tasks: The Rubric Tree Trick
  41. 081
    When Reasoning Models Decide Before They Think: Detecting and Fixing Premature Confidence
  42. 078
    Training a Markdown File: When LLM Self-Improvement Borrows the Discipline of Neural Net Training
  43. 074
    How a Fifteen-Hundred-Dollar Training Run Matched Llama and Gemma on Reasoning
  44. 073
    When Three LLMs Talk to Each Other, Their Ideas Quietly Stop Moving
  45. 068
    The OS Trick That Makes Tree Search Practical for Coding Agents
  46. 064
    When Agent Memory Stops Being a Database and Starts Being a Skill
  47. 060
    When Splitting One Model Across Three Agents Doubles Its Accuracy
  48. 053
    An AI Agent Swapped In Focal Loss And Beat A Human-Tuned Training Script
  49. 048
    How a 30B Open Model Reached Olympiad Gold With the Right Recipe
  50. 047
    When Agent Benchmarks Lie: The Harness Problem in Open-Source AI
  51. 043
    When 'This Is False' Doesn't Stick: Why Models Learn the Lie Anyway
  52. 041
    When the Iteration Teaches the Model to Skip the Iteration
  53. 040
    Two Frozen Models Learn to Whisper: Coupling Through Hidden States
  54. 039
    When Smarter Agents Get Fooled by Three Extra Nodes in a Database
  55. 038
    How LLMs Get Persuaded: One Attention Head, A Tetrahedron, And A Single Dial
  56. 037
    Why Hallucination Detectors Miss Stale Facts: A Geometric Story About What Models Know But Don't Say
  57. 036
    Sparse Attention Was the Wrong Frame. Treat It as Geometry Instead.
  58. 034
    Catching Multi-Agent Deadlocks Before Deployment With a 40-Year-Old Tool
  59. 031
    When Your AI Assistant Won't Let Go of Old Facts About You
  60. 030
    Why Your AI Agent Won't Stop Working — and Each Model Falls for a Different Trap
  61. 029
    Why Forty-Eight Percent on FrontierMath Isn't the Real Story in DeepMind's New Math Paper
  62. 028
    Teaching a Model to Hire Copies of Itself: Recursive Agent Optimization
  63. 027
    When AI Agents Build the Serving Stack: A Bet on Bespoke Infrastructure
  64. 022
    Training the Model Spec Directly: An Alignment Lever Aimed at the Say-Do Gap
  65. 019
    When the Best Reward Model Trains the Worst Policy: Inside EvoLM
  66. 018
    Language Models Compute the Rational Move, Then Override It
  67. 016
    Why Your Coding Agent Stalls While the GPU Runs Hot
  68. 010
    When Reward Climbs But Reasoning Goes Generic: Diagnosing Template Collapse in Agentic RL
  69. 008
    Why Long-Horizon AI Agents Get Stuck, and a Milestone-Based Fix That Helps
  70. 006
    What Happens Inside Claude When It Decides to Blackmail Someone
  71. 005
    Why a Debugger Designed for Humans Is the Wrong Tool for an AI Agent
  72. 003
    How to Pick the Best of Sixteen Coding Agent Rollouts

Related concepts

Activation Steering Amortized Inference Classifier-Free Guidance LLM Inference Systems LLM Serving Parallel Sampling Systems for ML Test-Time Auditing Test-Time Compute

Related terms

KV cache latency test-time scaling