Concept · 20 episode(s)

Inference Cost

← all concepts

Definition

Inference cost is the price — in dollars, watts, or latency — of running a model once it’s trained. At scale it dwarfs training cost, and it’s often the binding constraint on which features can actually ship.

Episodes covering this

  1. 198
    The Model That Knows the Answer and Can't Say It
    Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale
    Gollapudi, Gupta, Singhal et al. · UC Berkeley·17 min·Jul 03, 2026
  2. 179
    How DeepSeek Made One User Faster Without Slowing Down the Crowd
    DSpark: Confidence-Scheduled Speculative Decoding with
    XinCheng, XingkaiYu, ChenzeShao et al. · Peking University / DeepSeek-AI·23 min·Jun 27, 2026
  3. 170
    When a One-Liner Beats Your Agent's Clever Verification Logic
    Bayesian control for coding agents
    Papamarkou, Smirnov, Mazanov et al. · PolyShape / National Technical University of Athens·26 min·Jun 24, 2026
  4. 169
    Why Better Bug Reports Can Make AI Coding Agents Worse
    SHERLOC: Structured Diagnostic Localization for Code Repair Agents
    Tamoyan, Narenthiran, Arakelyan et al. · NVIDIA / TU Darmstadt·24 min·Jun 24, 2026
  5. 159
    Can a Coding Agent Run Its Own Robot Experiments Overnight, With No Human Resetting the Scene?
    ENPIRE: Agentic Robot Policy Self-Improvement in the Real World
    Xiao, Xie, Zhang et al. · NVIDIA·23 min·Jun 19, 2026
  6. 142
    Training a Tiny Model to Run the Plumbing Between an Agent and the World
    HarnessBridge: Learnable Bidirectional Controller for LLM Agent Harness
    Wang, Wang, Taylor et al. · University of California·24 min·Jun 12, 2026
  7. 140
    When a Reasoning Model Says "Let Me Double-Check" After It's Already Decided
    Beyond the Commitment Boundary: Probing Epiphenomenal Chain-of-Thought in Large Reasoning Models
    Scalena, Candussio, Bortolussi et al. · University of Groningen / University of Milano-Bicocca·27 min·Jun 12, 2026
  8. 116
    Why Streaming Half a Reasoning Chain Beats Sending the Whole Thing
    Streaming Communication in Multi-Agent Reasoning
    Yang, Xu, Wang et al. · HKUST (GZ)·26 min·Jun 04, 2026
  9. 115
    Teaching a Phone Agent to Reason Silently, And Keeping It Honest
    MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Models
    Yang, Hu, Hao et al. · Beihang University·24 min·Jun 04, 2026
  10. 097
    Same Tokens, Same Cost, Wildly Different Results: What Actually Scales in AI Agents
    Scaling Laws for Agent Harnesses via Effective Feedback Compute
    Zhang, Wang, Xu et al. · Harbin Institute of Technology·25 min·May 29, 2026
  11. 090
    How MiniMax-M2 Bets That Sparsity Plus Verifiable Rewards Can Match Frontier Agents
    The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
    MiniMax · MiniMax·28 min·May 27, 2026
  12. 085
    Why Long-Context Models Might Need Compute, Not Capacity, Before Eviction
    Language Models Need Sleep
    Lee, McLeish, Goldstein et al. · Carnegie Mellon University·24 min·May 26, 2026
  13. 077
    Reading a Model's Confidence Curve to Decide When Chain-of-Thought Is Worth It
    When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions
    Xia, Wang, Tang et al. · State Key Laboratory of General Artificial Intelligence·22 min·May 25, 2026
  14. 064
    When Agent Memory Stops Being a Database and Starts Being a Skill
    Auto-Dreamer: Learning Offline Memory Consolidation for Language Agents
    Ye, Liu, Wang et al. · University of Illinois Urbana-Champaign·30 min·May 22, 2026
  15. 063
    Why Web Agents Are Slow: A Compiler-Style Fix for Computer-Use Latency
    Agent JIT Compilation for Latency-Optimizing Web Agent Planning and Scheduling
    Winston, Wang, Mirhoseini et al. · Stanford University·26 min·May 21, 2026
  16. 041
    When the Iteration Teaches the Model to Skip the Iteration
    Solve the Loop: Attractor Models for Language and Reasoning
    Fein-Ashley, Rashidinejad · University of Southern California·30 min·May 13, 2026
  17. 036
    Sparse Attention Was the Wrong Frame. Treat It as Geometry Instead.
    Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
    Dehghankar, Asudeh · University of Illinois Chicago·24 min·May 11, 2026
  18. 035
    Why Frontier Agents Ask for Clarification at Exactly the Wrong Moment
    Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?
    Gulati, Gupta, Lumer et al. · PricewaterhouseCoopers U.S.·29 min·May 11, 2026
  19. 033
    Echo: The Paper Arguing You Never Needed a KV Cache for Retrieval
    Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators
    Sridhar, Johansen · California·24 min·May 11, 2026
  20. 005
    Why a Debugger Designed for Humans Is the Wrong Tool for an AI Agent
    Empowering Autonomous Debugging Agents with Efficient Dynamic Analysis
    Xiang, Xu, Chu et al. · Southern University of Science and Technology·22 min·May 01, 2026

Worth reading next

Papers we haven't done a deep dive on yet, but would recommend on this topic.