Definition
Long-context models accept and reason over very large inputs — hundreds of thousands or millions of tokens. The headline number on the spec sheet is rarely the same as the effective context: useful long-context work involves architecture, training, and serving choices all the way down.
Episodes covering this
Worth reading next
Papers we haven't done a deep dive on yet, but would recommend on this topic.
- Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
- THREAD: Thinking Deeper with Recurrent Multi-Hop Reasoning
- SnapKV: LLM Knows What You are Looking for Before Generation
- Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
- Larimar: Large Language Models with Episodic Memory Control
- ReadAgent: A System for Agent-Based Long-Context Reading Tasks
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
- Lost in the Middle: How Language Models Use Long Contexts