Language-model teams learn to coordinate through voluntary messages
s$^3$: Simple Scaling of Swarms via Cooperative Multi-agent RL · arXiv:2611.08615
In plain words
This paper introduces s³, a way to train teams of language models to work together without assigning them jobs or telling them how to communicate. Each model works on its own copy of a task and can choose when to message teammates or read their messages.
In tests about sharing private numbers and searching rooms, teams trained with s³ often did better than the study’s comparison methods. For example, on the number-sharing task, 93% of four-agent team attempts succeeded. The study tested only two tasks with small models.
The research
When a task is split across agents, assigning roles and deciding what they should share can shape whether teamwork helps.
These results make it plausible that training can teach a language-model team to organize its own communication, with measurable gains on private-information sharing and room search in the tested tasks.
s³ trains language-model teams to coordinate through voluntary messaging, using a shared reward so communication and task actions are learned together.
On held-out tasks, four-agent s³ teams assembled private numbers correctly in 93% of attempts, compared with 35% for online imitation and 13% for communicating agents trained with individual rewards. Success required any one agent to submit the exact sorted list.
In a separate search task, the communicating team averaged a 7.3-point success advantage over six solo-trained agents at six agents, after correction for multiple comparisons. Together, these results show coordination gains on both tested tasks, with the search advantage established at six agents.
The results come from two constructed tasks, small language models and one training run per condition. More runs, larger models and broader tasks could change how far the findings extend.
What the evidence shows
- Coordination without assigned roles
s³ is a cooperative reinforcement-learning method that trains teams of language-model agents to organize their work through voluntary messages. The agents have no assigned roles or fixed communication structure; each works on a separate copy of the task and chooses when to contact teammates.
The paper asks whether this kind of learned coordination can help with two challenges: combining private information and dividing a search.
- Sharing private facts
On the four-agent private-number task, the evaluator counted success when any agent submitted the exact sorted list. s³ succeeded on 93% of held-out attempts, versus 35% for online imitation, and also outperformed the individual-reward control.
Because the controls used different training objectives, the comparison shows an advantage over both approaches but does not isolate why.
- A bounded search advantage
The search task asks agents to find an exit in an unknown tree of rooms, where each agent explores its own copy. At six agents, the communicating team’s success rate was 7.3 percentage points above six independent copies of a solo-trained agent on held-out trees.
This difference remained significant after correction for multiple comparisons. At four agents the comparison was not significant after correction, so the supported advantage is specific to the six-agent result.
- What the controls and scaling show
A separate four-agent search test removed messaging while retaining the shared team reward; success fell by 19.27 percentage points. This supports a role for messaging in exploration under that training setup.
In private-number aggregation, s³ also retained an advantage over online imitation when evaluated with eight agents despite training with four. These tests suggest some coordination carries to larger teams, while the results remain limited to the two constructed tasks and small models.
Deep
- Original paper authors
- Kunal Jha, Gonçalo Faria, Shuyue Stella Li, Scott Geng, Faeze Brahman
- Submitted to arXiv
- License
- Not specified by source