Training updates reflect confidence in verified answers
SCARL: Sequence Confidence-Aligned Reinforcement Learning · arXiv:2610.40407
In plain words
When language models learn from answers that can be checked, answers with the same result can receive the same training update, even if one was a shaky success and another a confident mistake.
SCARL changes those updates using how confident the model was in each answer compared with its other answers to the same question.
In tests on competition-math problems, Qwen3-4B-Base trained with SCARL had higher average coverage than with DAPO alone: coverage estimates whether at least one of 32 sampled answers solves each problem. The finding is an average across five tests, not a result for every problem.
The research
When a group of answers receives reward-based updates, should equally scored answers be treated alike if the model was confident about one and uncertain about another?
Standard group-relative methods ignore that confidence, so a correct but fragile answer and a confident error can receive the same sequence-level advantage as other answers with the same reward.
SCARL reshapes group-relative training updates using each answer’s confidence and verified outcome, while leaving the estimator’s reward and clipping rules intact.
The central question is whether answers with the same reward should receive the same training update when the model’s confidence in them differs.
Standard group-relative methods ignore that confidence, so a tentative success and a confident error can receive the same sequence-level advantage as other answers with the same outcome.
SCARL compares each answer’s confidence with other answers to the same prompt and with its verified outcome, then uses that gap to reshape the advantage. In the main comparison, evaluators checked sampled answers with rule-based verifiers on five competition-math tests.
For Qwen3-4B-Base, SCARL with DAPO raised average Pass@32 to 69.5% from 67.1%; this estimates how often a problem gets at least one correct answer among 32 samples, averaged over problems and tests.
The experiments cover competition mathematics with binary, rule-based rewards and language models up to 4 billion parameters. They do not establish how SCARL performs in other domains or with other reward types.
What the evidence shows
- The confidence mismatch
Sequence Confidence-Aligned Reinforcement Learning (SCARL) is a method for adjusting the advantages that group-relative reinforcement learning assigns to sampled answers. The paper focuses on a mismatch: answers with the same reward can receive the same advantage even when the model produced one confidently and the other tentatively.
With binary, rule-based rewards, that can treat a fragile correct answer and a confident error like other answers with the same outcome.
- A broader problem score
The evaluators tested models on five competition-math problem sets, which ask for mathematical answers checked by rule-based verifiers. Their Pass@32 measure estimates the share of problems with at least one correct answer among 32 samples per problem, then averages across problems and tests.
For Qwen3-4B-Base, DAPO with SCARL averaged 69.5%, compared with 67.1% for DAPO alone under the shared training and evaluation recipe.
- Transfer with exceptions
The evaluation paired SCARL with four group-relative training methods across four model backbones. Average Pass@32 increased in 14 of the 16 method–backbone pairings compared with the matching method without SCARL.
The two exceptions were SAPO paired with Qwen3-1.7B-Base or Hunyuan-1.8B-Pretrain; each average fell by 0.2 percentage points. The pattern supports transfer across many tested pairings, not every pairing or every test.
- Testing the ingredients
In an additional component test using Qwen3-1.7B-Base and GSPO, removing the outcome target brought average Pass@32 close to the base estimator’s score, while removing class-wise normalization produced a much lower score.
These results suggest that both the outcome alignment and normalization contribute in this tested setup. They help explain the main result, but the ablation covers only one backbone and estimator.
- A qualified mechanism
Training measurements provide a separate check on the proposed mechanism. For all four estimators on Qwen3-4B-Base, entropy with SCARL was higher than with the matching base estimator from logged step 12 onward.
Yet entropy still collapsed under GRPO, though more slowly. So the measurements are consistent with SCARL slowing the loss of exploration, but do not support a claim that it prevents collapse across all estimators.
Deep
- Original paper authors
- Zhenghong Ke, Jingbo Zhou, Hua Wu
- Submitted to arXiv
- License
- CC BY 4.0