Scoring options separately keeps choices order-stable
In plain words
The paper introduces Prism-1, a way for a language model to choose from options and return a probability for each in one pass. It judges each option in its own block, without letting options affect one another, then turns the scores into probabilities.
The authors tested two models on five choice-task sets. Prism-1’s answers did not change when a list was shuffled, while trained single-choice prompting changed some answers; average accuracy remained similar.
The research
When another part of a system creates or rearranges a list of choices, can a language model avoid changing its answer just because of that order, and can adding an irrelevant choice leave the odds among the original choices intact?
The paper proves the scoring form that meets both consistency rules, then builds Prism-1 to score each option independently and turn those scores into choice probabilities.
With the same data and training budget, the authors compared Prism-1 with single-choice prompting; their decision reads and training targets differed. Across five task-set averages, ordinary accuracy was similar, with no significant pooled difference.
Order-robust accuracy, counting only answers correct in both the original and one shuffled listing, favored Prism-1 on both models. Its answers never changed under that shuffle, supporting the claim that independent scoring preserves order consistency without a broad accuracy penalty.
The empirical evidence covers two open models, sets of at most 20 options, and one to three training runs per configuration. The construction’s guarantee is exact only up to numerical rounding, which can tip near-ties.
Analysis
Deep
- Original paper authors
- Yao Long Teng, Jianguo Huang, Yanchen Deng, Jiayi Cai, Bo An
- Submitted to arXiv
- License
- CC BY 4.0