GenAI Radical Fragility

Small prompt or model changes produce unforeseeable behavioural shifts, so agent harnesses need continuous offline regression—not one-time validation.

GenAI systems are “grown, not built”: builders cannot precisely predict how a prompt tweak, training change, or silent cloud model swap will move behaviour. Bach & Bolton call this radical fragility — a massive regression-testing problem rooted in the model, not only in application code. Stochastic output and chaotic sensitivity (a path that works on one dataset fails on a slight variant) compound the issue.

Harness implication: treat every prompt, tool schema, and model pin as a regression surface under harness drift awareness and offline prompt evaluation. Prefer per-task offline harness tests that re-run on provider updates. Do not ship agent autonomy upgrades without re-checking the suite — see genai use safety modes.

---

Sources

• Taking Testing Seriously (Bach & Bolton), ch. 7