Demonstration Mapping Ablation

Randomizing demonstration labels tests whether few-shot gains come from the intended input-output rule or from other prompt signals.

To test whether an in-context learning prompt teaches the intended rule, replace each demonstrated output with a random member of the true output set while preserving the inputs, output distribution, and pair format. If performance remains close to the gold-demonstration condition, the gain cannot primarily come from learning the demonstrated mapping.

Across natural-language classification and multiple-choice tasks, this ablation often preserved most few-shot improvement, while model and dataset exceptions remained. The result is a diagnostic, not a license to use wrong examples: synthetic mappings, open-set generation, and tasks whose labels lack strong pretrained associations may behave differently. Run the ablation on the actual model and task, alongside separate tests of demonstration input distribution, demonstration output space, and paired demonstration format.

---

Sources

• Min et al. (2022), full paper, pp. 1–19