Glossary · Term

embedding

← all terms

Definition

Plain language

A list of numbers that represents the meaning of a word or sentence, so the computer can compare meanings mathematically.

As stated in the literature

A dense vector representation of a discrete input (token, sentence, image patch), learned so that semantic similarity corresponds to vector closeness.

Also called: embeddings

Why it matters: Embeddings are the bridge between symbolic data and the geometric world where neural networks actually do their work, underlying everything from search to retrieval-augmented generation.

For example, the words 'king' and 'queen' end up with very similar embeddings, while 'king' and 'broccoli' point in very different directions.

Heard on the show

“… At the raw embedding layer, before any processing, the probe finds essentially nothing — the signal only emerges deeper …”
Episode 204 — The Length Estimate Hiding Inside a Word-by-Word Model

Mentioned in 17 episodes

  1. 204
    The Length Estimate Hiding Inside a Word-by-Word Model
  2. 198
    The Model That Knows the Answer and Can't Say It
  3. 157
    When an AI Coding Agent Drives a Phone Through the Terminal, No Screen Needed
  4. 151
    Why More Experience Made This AI Agent Worse, And How to Fix It
  5. 132
    The Agent Failed — But Did the Instructions Deserve to Be Followed?
  6. 127
    What Diffusion Language Models Were Missing: A Map, Not an Algorithm
  7. 114
    Agents That Rewrite Their Own Weights Instead of Just Taking Notes
  8. 110
    How an Agent Got 44 Points Better by Mining Its Own Scratch Paper
  9. 106
    Giving Agents a Notebook Instead of New Weights: How ExpGraph Lets Frozen Models Learn
  10. 102
    How to Catch an AI Attack That No Single Conversation Reveals
  11. 085
    Why Long-Context Models Might Need Compute, Not Capacity, Before Eviction
  12. 073
    When Three LLMs Talk to Each Other, Their Ideas Quietly Stop Moving
  13. 067
    An AI Just Solved a 1996 Erdős Problem—and the Simplest Agent Won
  14. 041
    When the Iteration Teaches the Model to Skip the Iteration
  15. 039
    When Smarter Agents Get Fooled by Three Extra Nodes in a Database
  16. 019
    When the Best Reward Model Trains the Worst Policy: Inside EvoLM
  17. 002
    An AI Ran a Real Optics Lab for 21 Hours and Found a Transformer-Shaped Pattern in Light

Related concepts

Contrastive Loss Dense Retrieval LIMIT Benchmark

Related terms

token