Training with direct budgets reduces judged harmful answers
Safety as a Budget: Calibrating the Control Channel of Constrained GRPO for Vision-Language Models · arXiv:2611.16628
In plain words
Vision-language models answer requests that may contain both words and pictures. Training them to avoid harmful answers can also make them refuse harmless requests. This paper introduces a way to set separate limits for harmful help and unnecessary refusal.
The method uses a response scorer to guide training, adjusting its probability levels so the limits track measured rates. In tests, an independent judge found fewer harmful answers on average with the method than with standard training, for one model and three training runs.
The research
Choosing a safety reward weight indirectly can leave a model at an unwanted balance: it may help with harmful requests or refuse legitimate ones. Direct limits for both rates make that tradeoff explicit. In the tested setup, harmful answers fell, and over-refusal stayed at 1–2% on 400 benign queries.
Two-channel constrained GRPO calibrates scorer probabilities for separate harm and refusal budgets while preserving raw scores for ranking responses during training.
The method was tested by training Qwen3-VL-2B-Instruct with separate harm and refusal budgets, then comparing it with standard GRPO without a budget.
An independent language model judge labeled held-out adversarial responses, and researchers averaged the harmful-response rate across three training runs.
The calibrated method averaged 4.8%, versus 8.9% for standard GRPO. This supports a lower judged rate in the tested setup; a separate test also found lower rates on attacks that hide harmful intent in images.
The main comparison covers one starting model, a particular training setup and judge-labeled prompts. A human check used one annotator and 200 sampled responses, so it offers limited additional validation.
What the evidence shows
- Set two rates directly
Two-channel constrained GRPO is a method for training vision-language models to maximize helpfulness while setting separate budgets for harmful assistance and refusals of benign requests. Standard GRPO combines safety and helpfulness through a reward weight chosen by repeated training and evaluation. That approach can leave the model at a tradeoff between them that was not directly chosen.
- Separate score roles
The method gives the scorer two roles: raw scores rank responses for the model update, while calibrated probabilities estimate event rates for the budget controller. A scorer that ranks responses well can still report probability levels that misstate the rate.
The controller uses those levels to adjust multipliers, which change the strength of each budget’s penalty.
- Keep budget pressure
The paper’s analysis identifies another failure: standardizing the combined penalized reward within each prompt’s response group can cancel the safety multiplier before it strengthens the model update. The method preserves that multiplier’s influence by leaving advantages unscaled or standardizing each objective before weighting it. This links the budget pressure to the gradient that updates the model.
- Lower rates in tests
The calibrated method had lower judged harmful-response rates on held-out adversarial prompts and on HADES attacks that hide harmful intent in optimized images.
The HADES result extends this pattern to a test not used in training or calibration, but both comparisons concern the tested model and setup.
- Keep the conclusion bounded
The comparisons support lower judged harmful-response rates for the tested model and setup, including on HADES, which was not used in training or calibration. They do not settle how the method performs with other models or evaluation conditions.
A human audit kept the model ordering, but it covered just 200 sampled responses labeled by one annotator.
Deep
- Original paper authors
- Bo Han
- Submitted to arXiv
- License
- arXiv non-exclusive