Definition
Plain language
When something helps in one situation and hurts in another, so its overall average looks like it does nothing.
As stated in the literature
The property that an intervention's effect varies in sign across contexts or subpopulations, so a near-zero average masks large opposing conditional effects; in agent-skill curation, over 90% of skills help on some task types and harm on others, making average-effect filtering useless.
Why it matters: It warns that judging an intervention by its overall average can hide large benefits and harms that cancel out, leading to bad keep-or-drop decisions.
For example, a coding shortcut that speeds up simple scripts but quietly breaks complex ones can look harmless when you only check its average effect.
Heard on the show
“They call it causal heterogeneity, which is a mouthful, but the idea is simple.”Episode 151 — Why More Experience Made This AI Agent Worse, And How to Fix It