Matched tests find constrained decoding need not hurt accuracy
Constrained Decoding: Harmful or Beneficial? Revisiting Its Impact on LLM Performance · arXiv:2610.35328
In plain words
Language models sometimes fail to put an answer in the required shape, such as a JSON object. This paper studies whether rules that limit the words a model can generate also make its reasoning less accurate.
The authors compare answers when the prompt and required format stay the same. In those comparisons, enforcing the format and ruling out answers that do not fit the task generally matched or improved accuracy.
The result applies to the models and tasks tested; it does not mean every kind of restriction helps.
The research
When a language model must return a structured answer, it may fail to follow the required format. Earlier comparisons also found lower accuracy for reasoning in structured formats, but changed prompts and formats at the same time. Does restricting the tokens a model can produce cause the accuracy loss?
The paper separates format, answer-validity and reasoning constraints, then tests whether restricting generated tokens changes accuracy under matched prompts and formats.
To tell whether token restrictions themselves reduce reasoning accuracy, the authors compare restricted and unrestricted generation with the prompt and required output format held the same. They score an answer as correct only when it exactly matches the extracted ground-truth answer.
Across the tested models and tasks, format and validity constraints match or outperform unrestricted generation. This suggests that lower accuracy across different answer formats does not by itself show harm from constrained decoding, because those comparisons also change prompts and formats.
The experiments cover specific models, prompts, formats and reasoning tasks; validity constraints were tested mainly on two letter-concatenation tasks. The theoretical result also depends on the constraint allowing the correct answer.
What the evidence shows
- Separate format from decoding
This paper presents a taxonomy and experiments on constrained decoding, which limits the tokens a language model may generate. The question is whether the lower accuracy sometimes seen with structured answers comes from those restrictions or from differences in prompts and required formats.
The distinction matters to interpreting comparisons: if several conditions change at once, an accuracy gap cannot identify which change accounts for it.
- A format gap has confounds
The authors compare JSON reasoning with approaches that let the model reason freely and format the answer afterward. Accuracy differs, but the approaches also use different prompts and expected formats.
This observation motivates a more controlled comparison, rather than establishing that constrained decoding itself caused the gap.
- Matched comparisons test the cause
In the more direct comparison, the prompt and expected output format are kept the same while constrained and unconstrained generation are compared. Accuracy means an extracted answer exactly matches the correct answer.
Across the tested models and tasks, format and validity constraints match or outperform unconstrained generation, which weighs against the claim that token restrictions inherently reduce accuracy in these settings.
- Different constraints behave differently
The paper separates rules that enforce structure from rules that also exclude answers outside a task’s permitted answer space. On two letter-concatenation tasks, adding validity rules generally gives more consistent accuracy gains over format rules alone.
Required reasoning patterns have a less uniform effect: they can help, but one tested model and math task showed lower accuracy when such a pattern was added.
Deep
- Original paper authors
- Ngoc Trinh Hung Nguyen, Alonso Silva, Armen Aghasaryan, Mehwish Alam
- Submitted to arXiv
- License
- CC BY 4.0