Glossary · Term

separator token

← all terms

Definition

Plain language

The little marker in the text that divides one part of the input from the next.

As stated in the literature

A delimiter position between segments (e.g. problem and solution); often accumulates aggregate representations, making it a natural readout site for whole-input state probes.

Also called: separator tokens

Why it matters: Because these positions tend to gather information about everything before them, they are convenient places to read off or intervene on a model's overall picture of the input.

For example, the marker sitting between a puzzle's description and the start of the solution is a spot where researchers often find a compressed summary of the whole problem.

Heard on the show

“That geometry is only readable at one specific spot — the separator token, the position sitting between the problem and the solution, where the planning has to happen.”
Episode 237 — The Model Built a Perfect Map of the Puzzle, Then Lost It

Mentioned in 1 episode

  1. 237
    The Model Built a Perfect Map of the Puzzle, Then Lost It

Related terms