Injection Defences Are Bypassable

Adaptive attacks bypassed 12 published prompt-injection defences at over 90% success and human red-teams always succeeded, so model-layer defences are one layer of protection, never the security boundary.

Evidence: weak

Many published defences against prompt injection work at the model layer: classifiers that flag suspicious input, prompt wrappers that tell the model to ignore embedded instructions, or fine-tuning for robustness. They tend to be evaluated against fixed attack sets. The question that matters for a deployed agent is whether they hold against an attacker who sees the defence and adapts.

Willison reports a paper titled "The Attacker Moves Second", written by authors from OpenAI, Anthropic and Google DeepMind, that tested exactly this. Adaptive attacks got past 12 published defences with success rates above 90%, and human red-teamers succeeded every time. The evidence here is weak in one specific sense: the paper is known here only through Willison's summary, so the experimental details and the exact defence list are unverified.

The direction of the finding is consistent with the rest of the agent-security literature. The design-patterns and CaMeL work starts from the premise that the model cannot be trusted once it has read hostile text (see untrusted input must not trigger actions), and Anthropic's containment guidance puts deterministic environment boundaries first and model-layer steering second (see contain at the environment layer first). The prompt-side techniques in prompt-level attack defenses and defensive prompt engineering, and ranking tool output low in the instruction hierarchy, still have a role as layers, as does scanning tool output before it reaches the model (inspect tool output before it enters context), but each should be expected to leak.

The consequence is that no prompt wording, system instruction or classifier in front of an agent should be treated as what keeps it safe. Its safety has to come from what it is structurally able to do, for example by holding at most two of the risky capabilities named in the agents Rule of Two, backed by the containment controls in agent system-level defenses.

Sources

Simon Willison, 2 Nov 2025, reporting 'The Attacker Moves Second' (authors from OpenAI, Anthropic and Google DeepMind) — https://simonwillison.net/2025/Nov/2/new-prompt-injection-papers/