Textual Optimizer Momentum

Keep a window of past variable values in the optimizer context so updates see trajectory history, not only the latest textual gradients.

Scalar SGD momentum mixes past gradients numerically. TextGrad’s momentum instead injects past iterations of the variable into the optimizer prompt (e.g. momentum_window=3 → <PAST_ITERATIONS>…</PAST_ITERATIONS>). The rewriter sees how the prompt or solution evolved, which can reduce oscillation when textual gradients conflict across batches.

Treat the window size as a harness knob under textual gradient descent: too small forgets useful history; too large burns context and may anchor on early bad variants. Combine with natural-language optimizer constraints when format must stay fixed across iterations.

---

Sources

• TextGrad (Yuksekgonul et al.), ch. 7