Definition
Plain language
An older collection of questions and model answers used to study which model should answer what.
As stated in the literature
Routing dataset with a more heterogeneous model pool; used as a replication setting where benchmark-based routing outperforms difficulty-based routing.
Why it matters: Testing a routing idea on a second, differently assembled dataset is how you find out whether it generalizes or just fit the quirks of the first one.
For example, researchers can replay the same question through this stored set of model answers to check whether a routing rule would have picked the right model.
Heard on the show
“They also replicate on SPROUT, an older dataset with a more varied set of models, and there, routing by benchmark beats routing by difficulty.”Episode 286 — Six LLM Routers Tested, And None Beat a Weighted Coin Flip