Definition
Plain language
Testing whether a lesson learned in one setting carries over to a totally different one.
As stated in the literature
Evaluation on held-out domains absent from the training distribution, used to distinguish a learned general policy from domain-specific memorization.
Also called: transfer study, four-domain transfer experiment
Why it matters: It separates a system that genuinely learned a principle from one that merely memorized the right answers for the situations it was shown.
For example, training a system to be cautious about stock predictions and then testing whether it is equally cautious about medical questions it never saw in training.
Heard on the show
“These next commitment rates come from a later four-domain transfer experiment, not from the scrambled-dashboard study.”Episode 251 — When a Fake Dashboard Makes an AI Agent Just as Confident