Solo scores often miss which models help a team
In plain words
Language models are often judged by how well they answer alone, even when they will work with other models.
This paper introduces a way to measure whether a model helps the group, then a training method called compositional reinforcement learning (C-RL) that rewards the answer made by the whole group.
The researchers found that solo scores usually did not predict a model’s contribution. In tests with one trained model and three fixed collaborators, C-RL improved group scores over the untrained model across all 24 tested task-and-collaboration settings.
The research
When language models work together, how can researchers tell which participant actually helps the system? A solo test measures a model answering alone, yet the research question is whether that model improves the shared answer when it joins other models.
Compositional reinforcement learning trains a model inside a collaboration and rewards the team’s outcome, aligning learning with its contribution to the shared result.
The paper estimates a model’s contribution by comparing a collaboration’s score with all models present against its score after one model is removed. It also proposes compositional reinforcement learning (C-RL), which trains a model inside the collaboration using the team’s outcome as its reward.
Across three protocols and eight tasks, C-RL improved on the untrained model in all 24 settings and scored highest in 21. This supports training for the shared result: solo-reward training sometimes raised a model’s solo score while lowering the team’s score.
The comparisons cover the reported models, collaborators, tasks, and collaboration protocols, and the source does not establish that every method began from the same checkpoint. Results involving proprietary APIs may shift when providers update their models.
Analysis
Deep
- Original paper authors
- Shangbin Feng, Yulia Tsvetkov, Jan Kautz, David Acuna
- Submitted to arXiv
- License
- CC BY 4.0