A calibrated reward helps models follow rules and answer well
In plain words
A response can obey every added rule, such as a required format, yet still fail to answer the request. This paper introduces CaLIF, a way to train language models that rewards rule-following while checking whether an answer remains plausible given the request alone.
Across five model configurations, CaLIF raised average rule-following scores above the base models while keeping answer-quality ratings near the base average. The result is limited to the models and tests the authors studied.
The research
When a model satisfies an output rule, such as a required format, how can we tell whether it still answered the request? A rule checker can give full credit even to an inadequate answer, so training against that score alone may reward the wrong behavior.
CaLIF combines rule-check rewards with a bounded check against the model’s earlier answers, aiming to improve constraint following while preserving answer quality.
The authors compared five model configurations trained with CaLIF or verifier-only IF-RLVR on IFBench, which scores prompt-level compliance with added output rules.
CaLIF averaged 34.64/100 for rule satisfaction, above the base models' 26.98, while answer quality averaged 4.59/10, near the base average of 4.51.
IF-RLVR scored higher on rule satisfaction, 44.93, but lower on answer quality, 2.96. This pattern supports the authors' conclusion that CaLIF can improve rule following while preserving answer quality better than verifier-only training in these configurations.
The evidence covers five model configurations, the reported training data, and these evaluation settings. It does not establish that CaLIF will preserve answer quality for other models, training data, or tests.
Analysis
Deep
- Original paper authors
- Sangyeon Yoon, Wooseok Seo, Hamish Ivison, Hyesoo Hong, Donghoon Kim
- Submitted to arXiv
- License
- CC BY 4.0