Jailbreak patches can also suppress benign prompts
In plain words
A model can give a harmful answer to a particular trick prompt. A patch tries to make it refuse that prompt, but it might also make the model refuse harmless prompts that look similar.
The paper introduces a collection of confirmed model failures and a test of nearby harmful and harmless prompts. Across four models, some patches improved refusal on harmful variations while also making the models refuse more harmless neighbours, showing why both outcomes need checking.
The research
When a model is patched to refuse a harmful prompt, does the repair also stop nearby harmful variations while leaving similar benign prompts answerable?
Aggregate safety and capability measures may not reveal either kind of local spillover, so the paper asks how to evaluate the repair around the failure itself.
The paper introduces model-specific failure targets and a local test that scores harmful correction separately from benign-prompt preservation.
The authors apply four activation-steering methods to each of four models, using 50 confirmed failures specific to that model. PatchBench-Local scores refusal on the source failure and harmful variants separately from non-refusal on nearby benign prompts.
Results show these outcomes can diverge: on Gemma, AST and AdaSteer had similar harmful-correction scores (87.1 and 87.6 out of 100), but AST preserved far fewer benign responses (28.4 versus 86.7). So stopping nearby harmful answers alone does not establish a selective repair.
PatchBench prioritizes clear, actionable failures and does not cover every jailbreak strategy, harmful domain, or model behaviour. Its local scores describe generated prompts around isolated failures; they do not certify broad robustness or task-specific utility.
Analysis
Deep
- License
- arXiv non-exclusive