Instruction vs Demonstration Optimization

Instruction-only optimizers rewrite system prompts; demonstration bootstrappers add few-shot traces — complementary levers that can be combined.

Two common prompt optimization tooling strategies attack different parts of the prompt:

• Instruction optimization (e.g. TextGrad on a system prompt with requires_grad) rewrites natural-language guidance while keeping zero demonstrations — inference stays short.
• Demonstration bootstrapping (e.g. DSPy BootstrappedFewShot) mines few-shot traces that pass a metric and packs them into the prompt — often stronger format teaching via in-context learning, at higher token cost per call. This is one instance of DSPy module parameterization, where demonstrations are an explicit, searchable parameter rather than a hand-picked constant, generated by the candidate-generation stage of DSPy's compiler.

On Big-Bench Hard Object Counting / Word Sorting and GSM8k, TextGrad’s instruction-only prompts matched or beat DSPy with up to eight demos; combining DSPy demos with a TextGrad-optimized instruction on GSM8k edged higher still. Treat them as complementary: optimize the instruction for reasoning habits (chain of thought, verify/re-check cues), then decide whether demos are worth the inference budget. Prefer strong-to-weak prompt optimization when the instruction search can use a stronger feedback model than the deployed reasoner.

---

Sources

• TextGrad (Yuksekgonul et al.), ch. 3 §3.3