A new test separates decision skill from reliability
SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models · arXiv:2611.15777
In plain words
SO-Arena is a test of models that make structured choices from information and a goal. It checks both fixed questions and tasks where a model chooses actions in a web or tool environment, using reference answers that researchers checked against what the model could see.
The test also checks whether choices stay consistent when options move, and whether a model notices when no offered answer is correct. In the evaluation, the System One models with the highest capability scores were not the strongest on the separate reliability measure.
The research
A strong task score can make a model seem dependable, but it does not show whether its choices stay consistent when options move or whether it notices that no answer fits. SO-Arena’s findings make reliability a separate question to consider alongside capability when judging decision models.
SO-Arena is a benchmark that tests System One models across decision tasks and separately measures whether their choices remain reliable as options change.
The authors compared 21 models on SO-Arena’s capability leaderboard and separate reliability tests. The leaderboard weights domains equally and emphasizes demanding tiers where available.
The System Two references led overall, while capability and reliability rankings diverged among tested System One models. This shows task performance alone does not capture how reliably those models choose.
SO-Arena’s difficulty tiers are relative to each domain, and two domains have no tiers. Its capability leaderboard emphasizes selected demanding tasks, so its aggregate score does not represent performance on every task.
What the evidence shows
- Capability needs a wider test
SO-Arena is a benchmark for assessing System One models, which turn supplied information and a decision goal into structured choices. It covers fixed questions as well as action selection in web and tool environments, and checks of whether recorded actions completed a goal.
This wider scope lets the evaluation ask whether a model can perform across different kinds of decisions and whether its choices remain reliable.
- The overall scores separate
The leaderboard’s selected task set puts the leading System One models below the System Two references overall. Since it weights domains equally and emphasizes demanding tiers where available, this is an aggregate gap on selected tasks, not a uniform gap on every task.
- Reliability changes the ranking
Capability and reliability are separate measures in SO-Arena, and their rankings diverged among the evaluated System One models. That matters because a high task score alone does not tell whether choices remain stable when options move or whether the model recognizes that none of the choices is correct. Reliability adds information that the capability leaderboard does not capture.
- Consistency is not correctness
The separate reliability score captures two behaviors beyond task performance: keeping the same choice when options are reordered and recognizing when no correct answer is offered.
Because stability tests consistency with the original answer, it cannot establish that the answer is correct; reliability therefore adds a distinct dimension to the capability score.
- Task type changes the gap
The selected task comparison shows that the gap depends on the domain: GLiDE was close to Luna in grounded decision tasks, while both tested System One models lagged in math, science, and code.
A separate control found that enabling explicit thinking helped StartLux Auto substantially in Math, illustrating that gains depend on both domain and inference condition.
Deep
- Original paper authors
- Lichen Li, Ziping Dong, Sicheng Feng, Qi Li, Xinchao Wang
- Submitted to arXiv
- License
- CC BY 4.0