Attack scores alone do not settle prompt-injection decisions
From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection · arXiv:2610.20907
In plain words
Prompt injection is text that tries to redirect an AI application with untrusted instructions. This evaluation study asks whether an attack score can help a system allow, block, or send an input to a reviewer.
The researchers froze the scoring and decision settings, then tested attacks and benign examples together. At the point where the systems missed no more than 1% of attacks in that test mix, they sent nearly all inputs to review and still blocked many benign ones. The result shows that the score alone does not decide how well the full system works.
The research
When an input may contain a prompt injection, can a probability score tell a system whether to let it through, block it, or send it to review?
A strong ranking might find attacks, but confidence that matches the actual input mix and a reviewer that handles deferred cases also shape the final decision.
This evaluation study tests whether attack probabilities can guide allow, block, and review decisions in a frozen gate-and-reviewer pipeline.
Researchers froze attack-scoring gates, their allow/block/review settings, and a language-model reviewer before testing 47,611 examples from four prompt-injection collections. Only deferred inputs went to the reviewer for a final decision.
On this mixture, reaching an observed miss rate of at most 1% of attacks required sending 91.89% to 92.55% of all inputs for review; 40.55% to 45.84% of benign inputs were still blocked.
This shows the gate score alone does not determine false positives; reviewer behavior and decision settings matter too.
The confirmatory cohort is a deliberately constructed mixture with 88.2% attacks, so its pooled rates do not predict natural deployment prevalence. The study reports point estimates without confidence intervals and measures detection and routing, not downstream agent compromise or harm.
What the evidence shows
- What a detector score must do
Prompt injection is untrusted text that tries to redirect an AI application. This paper is an evaluation study of whether attack-probability scores support decisions to allow, block, or defer an input to a reviewer.
A high score may help rank attacks, but its confidence also has to match observed outcomes, and the final decision depends on what happens to deferred cases. The study tests this question across a mixed prompt-injection workload.
- How the comparison was frozen
To keep confirmatory outcomes from shaping the tested system, the authors chose prompts, model settings, score adjustments, reviewer cutoff, and gate intervals on separate data, then froze them. Each gate allows, blocks, or defers an input; only deferred cases reach the fixed language-model reviewer.
The final test combines 47,611 examples from four preselected collections, but 88.2% are attacks, so this deliberately assembled mixture is not a natural prevalence estimate.
- Ranking favors different models
To compare attack ranking, researchers calculate AUROC, a 0-to-1 measure of how often an attack scores above a benign example across possible cutoffs; 1 represents perfect ordering. On the pooled examples, the language-model judge scored 0.897, Jev 0.871, and the Prompt Guard 2 86M and 22M classifiers 0.757 and 0.714.
The measure is calculated over pooled examples, not averaged across collections, and the comparison does not isolate training objectives.
- Confidence tells a mixed story
Calibration asks whether assigned probabilities match how often attacks actually occur; for example, inputs scored 0.8 should be attacks about 80% of the time. Jev’s native scores had lower expected calibration error than Prompt Guard 2 86M after probability adjustment on separate target data (0.541 versus 0.571), a nearly equal Brier score (0.440 versus 0.439), but worse negative log-likelihood (1.432 versus 1.104). Lower values are better for these measures, so the pattern is mixed.
- Review shifts the system trade-off
With the same reviewer, all three gates needed more than 91.8% of all confirmatory inputs sent for review to reach an observed system miss rate of at most 1% of attacks. At those points, 40.55% to 45.84% of benign inputs were still blocked.
This shows why the reviewer and decision settings matter alongside the gate score: calibration alone does not guarantee a useful security decision.
Deep
- Original paper authors
- Qianlong Lan, Anuj Kaul, Vinothini Pandurangan, Indranil Sanyal
- Submitted to arXiv
- License
- CC BY 4.0