Decision models struggle to recover every valid option
MMDM: Measuring Massive Multitask Decision Making · arXiv:2611.14309
In plain words
Massive Multitask Decision Making (MMDM) is a set of text questions for testing models that choose among options. A question gives a situation and asks the model to pick one answer, find every valid answer, or show how likely different outcomes are.
The researchers tested 28 models. On questions requiring the complete set of valid options, the top model got 36.81% correct, suggesting that choosing one answer does not capture every part of decision making.
The research
A model that picks one plausible answer may still miss other valid choices or misstate uncertainty. For structured decisions, that changes what a single accuracy score can tell us: MMDM separates choosing correctly from recovering a full set and representing probabilities.
Massive Multitask Decision Making (MMDM) tests whether text models can choose correctly, find every valid option, and represent uncertainty across decision tasks.
The evaluators tested 28 models by checking whether they ranked a correct single option first, placed every valid option above invalid ones, and matched reference probabilities. The highest complete-set accuracy was 36.81% of questions, while one model's single-choice accuracy ranged from 78.75% on World Knowledge to 47.95% on Constraint Planning. The results support treating decision quality as more than one score.
MMDM scores predictions for fixed text questions; it does not run the tools or workflows described, and prior exposure to underlying public source material may still occur.
What the evidence shows
- A broader decision test
Massive Multitask Decision Making (MMDM) is a benchmark for text models that assign probabilities to options in one step. It tests decisions where the answer may depend on a long situation, relationships among options, or constraints on which choices are valid.
Its purpose is to bring these demands together in a test that also uses newly constructed and recomposed questions.
- What existing tests miss
The motivation is that reuse of public test material can blur whether a model handles a decision pattern or has seen related material before. The paper also argues that existing evaluations do not consistently combine long context, large option lists, and logical dependencies.
MMDM addresses this gap with questions designed to test those demands together, while recognizing that source material may still be familiar.
- Different choices test different skills
The benchmark separates three kinds of performance: ranking one correct choice first, ranking the full set of valid choices above invalid ones, and matching probabilities across possible outcomes. In one model's single-choice results, accuracy was 78.75% on World Knowledge and 47.95% on Constraint Planning.
Since the model and measure stay the same across those categories, the contrast shows why familiar knowledge performance cannot stand in for constraint handling.
- Complete sets remain difficult
Even the highest complete-set score shows how often models missed at least one valid option, so single-choice success alone cannot capture that difficulty.
The results support the authors’ view that choosing one answer, recovering a full valid set, and representing probabilities measure different aspects of decision performance.
Deep
- Original paper authors
- Junhao Shen, Qing Zong, Yue Guo, Zhanqiu Zhang, Yiwen Guo
- Submitted to arXiv
- License
- arXiv non-exclusive