Definition
Plain language
Getting a better score by spending the expensive budget on the cheapest questions rather than the ones that actually need it.
As stated in the literature
Exploiting an accuracy-per-dollar objective by routing short-output (hence cheap) queries to the strong model, since each item contributes equally to accuracy but costs are per-token.
Why it matters: It means a routing system can post great scores on a cost-efficiency metric while actually serving users badly on the questions that most needed the strong model.
For example, a router that is graded on accuracy per dollar can learn to send every one-word trivia question to the expensive model — because the answers are short and cheap — while dumping the long, hard questions on the weak model.
Heard on the show
“The authors call it "accuracy hacking": call the stronger model when it's cheapest to do so.”Episode 286 — Six LLM Routers Tested, And None Beat a Weighted Coin Flip