Unlabelled Demonstration Baseline

Pairing representative unlabelled inputs with random valid labels provides a strong baseline for measuring what ground-truth demonstrations add.

A test-input-only zero-shot condition understates how much performance can be obtained without labelled supervision. Construct a stronger baseline from unlabelled in-distribution inputs, the true label set, and a recognizable input-output pair format, but randomize the pairings. This preserves the major structural and distributional signals while withholding the ground-truth mapping.

Compare this condition with gold demonstrations in offline prompt evaluation. A small gap suggests task location; a large gap indicates that correct pairings matter for this model-task combination. Call the condition zero-shot only when access to unlabelled task data is allowed by the evaluation contract.

---

Sources

• Min et al. (2022), full paper, pp. 1–19