A training reward balances requested content with answer quality
Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL · arXiv:2610.33512
In plain words
When training a model to answer open-ended questions, rewarding it for covering requested points can also encourage extra or incorrect details.
Fork is a reward method that asks a language-model grader to judge whether each substantive part of an answer is needed and factually correct, alongside whether the answer covers the requested points.
In a test of medical answers, Fork kept coverage similar to a standard reward while improving judged precision and preference for the trained model.
The research
When a model is rewarded for including every requested point, how can training keep it from adding redundant or incorrect material? Rubric coverage tracks whether requested elements appear, but does not by itself judge every extra sentence for necessity or accuracy.
Fork changes the training reward to combine rubric coverage with response precision: whether each substantive part is needed and judged factually correct.
The authors trained Qwen2.5-7B-Instruct using either standard weighted rubric coverage or Fork, then compared responses on the RubricHub-Medical test set. GPT-5.4, a separate reference grader, assessed rubric precision and recall and judged which response it preferred over the base model.
Fork’s test-set macro precision (the reported aggregate measure, whose detailed averaging rule is not specified in the paper) was 53.2%, compared with 25.5% for standard coverage, while recall was 32.5% versus 35.7%. Its win rate against the base model was 68.0% versus 3.1%.
The comparison suggests that adding a precision objective improved judged response quality and preference while keeping rubric coverage in a similar range.
The evidence covers Qwen2.5-7B-Instruct on the named Medical and Science data and tests, with results assessed by language-model graders.
Necessity and factual correctness are grader judgments, and the findings do not establish performance for models, domains, graders or judge prompts beyond those examined in the paper.
What the evidence shows
- Coverage can reward extra content
Fork is a reward method for training models to answer open-ended prompts against criteria called rubrics. The problem it addresses is that coverage rewards can encourage a model to include requested content while also adding unnecessary or factually incorrect material.
Fork adds a measure of response quality to coverage, so the training reward accounts for both what is included and whether the response units are useful and correct according to a grader.
- How Fork judges answers
To estimate response quality, Fork divides each answer into substantive units, roughly sentence-length. A language-model grader matches units to rubric criteria and shares each criterion’s credit among the units addressing it, which helps discount repetition.
The grader also labels units for necessity and factual correctness. Precision is the fraction of units judged both necessary and correct; recall is weighted rubric coverage. Fork combines these measures in its training reward.
- Coverage alone loses quality
The training-curve comparison on Qwen2.5-7B-Instruct used RubricHub-Medical and an in-domain test set. With standard coverage as the reward, recall quickly saturated while precision and preference against the base model declined and responses became more verbose.
Fork reached similar recall while precision and preference rose and responses were less verbose. This comparison shows the pattern Fork is designed to address in that setting.
- Precision rose with similar recall
On the RubricHub-Medical test set, GPT-5.4, the reference grader, reported test-set macro precision (the reported aggregate measure, whose detailed averaging rule is not specified in the paper) of 53.2% for Fork and 25.5% for standard coverage, while recall was 32.5% and 35.7%.
In separate comparisons against the base model, the win rate was 68.0% for Fork and 3.1% for standard coverage. Thus the reported preference and precision gains came with recall in a similar range for this comparison.
- Checks beyond the main comparison
A control evaluated rubric-based methods on RubricHub-Medical using the training grader, to examine the reward formulation apart from evaluation with a stronger unseen grader. Fork had the highest win rates among those methods in this control.
On HealthBench, an external test of medical answers, Fork’s win rate against the base model was 74.1% versus 7.1% for standard coverage, with similar rubric recall. Both findings are limited to their stated evaluation settings.