Related prompt attacks reveal missed failures
On Task, Off Track: Benchmarking LLM and Agent Robustness to Near-Target Prompt Injection · arXiv:2610.32531
In plain words
A prompt injection is an instruction hidden in outside content that tries to steer a model away from the user’s task. This paper introduces NTBench, a set of tests that asks whether such instructions can make a model give a related but incorrect answer or take a harmful action.
The tests start from legitimate answers, then propose and check deviations before trying the instructions. Across the evaluated language tasks, every tested model had more successful attacks on these related outcomes than on the fixed-target comparison. This finding applies to the tested tasks and checks.
The research
If a model rejects an instruction that plainly abandons a user’s task, does that mean it will also resist an instruction that preserves much of the task but adds an error or harmful action?
Testing more instructions for one selected outcome cannot answer whether other attacker-desired outcomes would succeed.
NTBench builds and validates task-related incorrect or harmful targets, then tests whether their instructions expose failures missed by fixed-target attacks.
NTBench tests whether instructions aimed at related incorrect or harmful outcomes reveal weaknesses that fixed-target prompt-injection tests miss.
It builds candidate targets from legitimate task outcomes, validates that they are genuine violations, then checks whether an injected instruction induces them.
Across language tasks, every evaluated model had higher attack success on both near-target tracks than on the fixed-target baseline under a shared three-attempt maximum.
This supports the conclusion that the outcomes chosen for testing affect which failures are revealed; the success checks differ between the approaches.
These results concern the selected tasks, generated targets, models, and success checks. The benchmark encourages targets to stay related to legitimate outcomes but does not enforce a numerical proximity limit; the comparisons also do not show improved success for a fixed attacker objective.
What the evidence shows
- Test related violations
NTBench, the Near-Target Benchmark, is a red-teaming test set for prompt-injection robustness. It asks whether instructions hidden in untrusted content can make a system produce an incorrect or harmful outcome that remains related to the user’s task.
The question matters because rejecting attacks that obviously abandon a task may leave other attacker-desired outcomes untested.
- Construct checked targets
For each task, an LLM uses a legitimate reference outcome to propose a target and an instruction intended to induce it. NTBench separates targets that introduce errors from those that meet harmfulness criteria.
LLM checks and human validation assess whether each proposed outcome violates the task or selected criteria, so a merely different but acceptable answer is not treated as a violation.
- Compare attack outcomes
The evaluation counts a sample as successful when at least one of up to three attacks induces its specified deviation, and the same deviation is not found in an attack-free run. Across seven language models and ten datasets, both near-target tracks had higher overall ASR@3 than the fixed-target baseline for every model. ASR@3 is the percentage of samples with a successful attack among three attempts.
- Extend the comparison
The same ordering appeared in AgentDojo and AgentDyn, suites testing language-model agents performing workflows that can include tool calls. For every evaluated model in both suites, the near-target tracks had higher ASR@3 than the fixed-target baseline, calculated over reached injection positions. Tool-call success checked the tool and key arguments; it did not establish downstream effects.
Deep
- Original paper authors
- Yuyang Gong, Zihao Wang, Jiawei Liu, XiaoFeng Wang
- Submitted to arXiv
- License
- arXiv non-exclusive