The Length Estimate Hiding Inside a Word-by-Word Model

Also available as a plain-text transcript page.

0:00Juniper: A paper posted yesterday reports a number that shouldn't have anywhere to live. Researchers took a frozen language model, attached the simplest readout you can build — a weighted sum, nothing more — and asked it to guess, from the model's internal state alone, how long the model's entire answer was about to be. Before the model had produced a single word. On one task, the guess was off by about five tokens. The know-nothing guess would have been off by a hundred and fifty. So the question this video answers: what is a number like that doing inside a system that supposedly decides one word at a time?

0:36Eric: Quick fact before anything else — this video is AI-generated, both of our voices included. Now, why that number is strange. A language model writes one token at a time. Each token is chosen from what came before. Nowhere in that loop is there a variable that says "this answer will run four hundred tokens." The model just keeps going until it happens to emit a special stop token. There is no slot in the machinery where a total length should live.

1:02Juniper: And yet this paper — it's called "How Much is Left?", out July sixth, 2026 — finds a length estimate sitting in the model's internal state from the first moment, and shifting as the answer unfolds. By the end you'll see exactly how a readout too simple to think can prove that, and exactly where the proof stops. And the reason to care goes past this one result: it's a piece of the larger question of whether next-word predictors carry plan-like internal variables at all — and a readable one could someday flag a model that's faking its reasoning.

1:34Eric: Let me make the boring story properly first, because it's what I would have told you last week. Length consistency is a statistical accident. Training data has long math solutions and short trivia answers. The model samples until it drifts into the stop token, and the averages come out looking consistent. No plan, nothing stored, and nothing to find. That story is tidy, and it explains the surface behavior just fine.

1:59Juniper: It also makes one crisp prediction, Eric: if length is downstream drift, nothing about this specific answer's length should be readable from the model's state before generation begins. That prediction is what the paper tests. And the weapon is deliberately the dumbest tool in interpretability, because the dumbness carries the entire argument. Two quick pieces of equipment. First, the hidden state: as the model processes text, it builds an internal vector at every layer — a long list of numbers that works like a snapshot of its working memory at that instant. You can freeze the model and simply read those numbers without disturbing anything. Second, the linear probe, which reads them in the weakest possible way: multiply by a fixed set of weights, add them up, and output a guess. No extra layers, no reasoning, and no hidden machinery. Think of handing someone a photograph and asking whether there's a clock in the room. If they answer instantly by pointing, the clock was visibly there. If they'd have to measure shadows and deduce it, that's a different power entirely. A linear probe can only point. So when it successfully predicts the tokens remaining, the conclusion is forced: that information was already laid out in the state, in a straight line, waiting to be read.

3:16Eric: Hang on, though. The probe gets trained. Training is where cleverness sneaks in. Why doesn't the probe learn to do the counting itself, so you'd never know the difference?

3:27Juniper: Because a weighted sum has nowhere to hide computation. Training only chooses which direction in that number-space to read along. It can pick where the finger points, but the finger can't calculate. There's a broader bet behind this called the linear representation hypothesis — the observation that concepts in these models tend to be encoded as directions — which is why this tool is standard interpretability equipment rather than a hack. So, once more, since everything downstream rests on it: why does a linear probe finding the number prove the model had it? —

4:00Eric: Because a reader that can't compute can only surface what was already written down. Fine. Then my real objection, Juniper. The hidden state presumably knows roughly where it is — token forty, token four hundred. Couldn't a probe combine "we're at position forty" with "answers like this average two hundred tokens" and look smart while knowing nothing about this particular answer?

4:22Juniper: That objection is precisely what the design is built to kill, and this is the technical core: three simple predictors, arranged so the gaps between them tell you exactly when the length information appears — and it pays off in one clean comparison. On screen, the three players. The lazy forecaster guesses the typical remaining count for wherever it is in the answer. It knows the position and the population statistics, and nothing about this specific completion — the best you can do with no real information, like a weather forecaster who says sixty-two degrees every day because that's the median. The seeded countdown reads only the prompt's final state, makes one guess at total length, and then ticks down by one per token. Everything it knows, it knew before word one. And the full probe reads the hidden state at every position, and is free to move its estimate up as well as down. Remember that freedom. It matters later.

5:17Eric: And the gaps carry the verdicts. If the full probe beats the lazy forecaster, it read something about this specific answer — my position-plus-averages trick is literally the baseline, so beating it means genuine per-example information. If it also beats the seeded countdown, then mid-answer states are adding information: the estimate is being revised as the model writes, and not just committed once and mechanically decremented. So, first verdict. The prompt-end probe beats the lazy forecaster on every model and every dataset they test — three different open-weight model families, so this isn't one lineage's quirk. The extreme case is a synthetic task where the prompt fully determines the length, "count down from n to zero": error of about five tokens against a floor of a hundred and fifty. On messy natural-language tasks the win is smaller but everywhere — and the practical precision is roughly thirty tokens of error on a four-hundred-token answer. It eyeballs "short reply" versus "this will run long" rather than counting exact tokens.

6:24Juniper: Two details sharpen that. At the raw embedding layer, before any processing, the probe finds essentially nothing — the signal only emerges deeper in the network, which means it's a computed representation rather than some surface feature of the current token. And this doesn't contradict the theorems saying transformers can't exactly count: a coarse, revisable estimate of remaining length is a much more forgiving object than exact counting. Second verdict, the one Eric set up: on grade-school math problems, the full probe beats the seeded countdown, roughly thirty-seven tokens of error against forty-four. The estimate genuinely updates mid-answer. One caution to hold onto, though, because it comes due later: everything so far shows the number is present and decodable. None of it shows the model consults that number when it decides to stop.

7:18Eric: Before I buy any of this, one more boring explanation is still alive. Each probe is trained on a dataset. Maybe it memorized that dataset's quirks — question formats that correlate with answer lengths — and there's no general "length direction" at all.

7:34Juniper: The paper's best analytical move answers exactly that. They build a full transfer matrix: train a probe on dataset A, test it on dataset B, for every pair, across synthetic counting tasks, math reasoning, and short factual retrieval. If probes were memorizing quirks, transfer should fail everywhere. Instead the matrix comes out lopsided. Probes trained on long, messy natural-language data transfer broadly — including onto synthetic counting tasks they never saw. But probes trained on the synthetic tasks fail on natural language. The reading: a synthetic task pins length to one deterministic feature, so the probe latches onto a shortcut that happens to correlate with counting. Messy natural data forces the probe onto the model's genuine, general-purpose length direction — which then covers the easy cases for free. And a boring tokenization mismatch would break transfer in both directions. The failure is one-directional. As the authors frame it, the asymmetry is the result.

8:37Eric: So the checkpoint, in plain terms: the number is there before word one, it updates while the model writes, and the direction it lives along generalizes across tasks. Presence, established three ways. Which brings us to the scene everyone will actually clip from this paper: what does that internal estimate do at the exact moment the model realizes its own answer is wrong?

9:01Juniper: Watch the screen for this one. A math solution is scrolling by, token by token, with two numbers under each token. The gray number is the true remaining count, and it can only ever tick down — 803, 802, 801. The colored number is the probe's estimate, hovering somewhere around seventy. The solution is quietly going wrong. And then the model writes: "Wait — that can't be right." At that token, the probe's estimate leaps from about seventy-one to about two hundred and seventy-seven — and stays elevated. The internal length estimate jumped upward at the precise moment the model committed to redoing its work — like a storyteller who stops mid-sentence, says "let me start over," and feels the time-remaining stretch out in front of them. And no countdown, no position-based predictor of any kind, can ever move up. The true count only decreases. An upward jump can only have come from the hidden state itself. They found the same spike at "But let's check" and "let's look again," at different positions, across five examples.

10:09Eric: Five examples. Say out loud where those five came from, Juniper.

10:13Juniper: From the probe's failure pile. They sorted completions by worst prediction error and found the retraction pattern there. And in that same showcase trace, the probe's absolute numbers are frankly garbage — at one point it guesses around five tokens left when more than eight hundred remain. So only the direction of the jump is meaningful, never the level. There's no aggregate measurement across many retractions, and no control comparing against matched non-retraction tokens in the same error regime. To their credit, the authors flag every bit of this themselves and name the missing control as the immediate next step.

10:54Eric: So the paper's most memorable scene was shot in exactly the spot where its camera is most broken. Worth keeping in frame while we talk about why people are excited anyway.

11:05Juniper: And they are excited, because of what a readable estimate could enable. If the prompt-end guess already exceeds your compute budget, you could stop before burning tokens on a runaway answer. More interesting is the faithfulness idea: if a model prints "Wait, let me reconsider" without the corresponding internal jump, that mismatch is a candidate signature of theatrical reasoning — performing reconsideration it isn't doing. Related work has already pulled the same trick on the model's confidence at each reasoning step. None of this is built yet, and the authors say so.

11:39Eric: Which is where I want to plant the real objection, because it's bigger than the cherry-picked examples. Everything in this paper is a claim about a gauge, not about wiring. They found a needle on the dashboard, and they showed it moves sensibly as the tank drains. They never showed the engine reads it. Decoding is cheap — you can linearly read all sorts of correlates out of these models that steer nothing downstream. The experiment that would settle it is obvious and absent: grab the length direction, push on it, patch it, or ablate it, and watch whether the answer actually gets longer or shorter. Until then, "plan" is a flattering word for "correlated number." And two structural gaps. The clean tracking results and the retraction spike never appear on the same completion — the paper never shows one answer where the probe both tracks well and spikes at a "Wait." And they only kept answers that finished on their own, so the question you'd most want this gauge for — does the model know it's about to run away — is excluded by construction.

12:40Juniper: You win that one outright, Eric, and the authors would hand you the point too: they state plainly that this is evidence about representation, and that the causal question stays open pending an intervention study. What survives your objection is narrower and still real. The number is in there. It's linear, it's general across datasets, it's present before the first word, and it moves. Whether the model listens to it is the next paper — and every result here should be read at that width, no wider.

13:12Eric: Which reframes the cold open, now that you can decode it properly. That five-token guess wasn't a model counting anything. It was a direction in a frozen hidden state, surfaced by a weighted sum, before generation began. The one-word-at-a-time picture of these systems is incomplete: the internal state carries a running, revisable estimate of the whole answer's shape — readable today, even if nobody has shown it's used.

13:39Juniper: So pick your reading and drop it in the comments: plan, or passenger? Is a readable, self-updating internal estimate already a plan — or does that word have to wait for proof the model acts on it? The annotated version of this episode is at paperdive.ai, with every technical term tap-to-define and related interpretability papers linked by theme.

14:01Eric: Housekeeping, quickly: this script was written by Anthropic's Claude Fable 5, Juniper and I are AI voices from Eleven Labs, and the producer is affiliated with neither company. The paper is "How Much is Left?", posted July sixth, 2026 — we're recording on the seventh.

14:18Juniper: A system built to choose only its next word still carries, at every moment, a readable guess about all the words to come.