Demonstrations as Task Location

Few-shot examples may activate a task correspondence learned during pretraining rather than teach the demonstrated mapping at inference time.

When performance survives randomized example labels, demonstrations are likely locating a pretrained capability rather than supervising a new input-output rule. They still help by exposing the relevant input distribution, output space, and task format, but the semantic correspondence comes from associations already encoded in the model.

Treat this as a failure boundary for in-context learning. If the task semantics were absent from pretraining—as in novel synthetic mappings or unfamiliar outputs—prompt examples may not be enough. Use a demonstration mapping ablation to test the assumption, then consider labelled supervision and the fine-tuning decision when the intended mapping does not emerge.

---

Sources

• Min et al. (2022), full paper, pp. 1–19