Shared question errors limit gains from larger model teams
Shared Item-Level Error Limits the Aggregation of Small Language Models in Fermi Estimation · arXiv:2611.16315
In plain words
Fermi questions ask for rough numerical estimates. This paper studies whether combining many language-model answers to the same question makes the estimate more accurate, as averaging can when mistakes differ and cancel.
The researchers separate a model’s recurring error on a question from the variation between repeated answers.
Across the tested open-weight models on the realFP set of Fermi questions, teams of 30 improved the error score only a little over one answer, consistent with shared question errors limiting the gain.
The research
The expected payoff from adding more model answers depends on whether their mistakes cancel. In these tests, most error variation was tied to the question, and larger teams barely improved estimates.
That changes what the team-size argument supports: adding answers alone may bring little accuracy gain when answers share question-level errors.
The study separates shared question-level error from answer-to-answer noise to explain why aggregating more model estimates brought little accuracy gain.
The researchers analyzed 2.51 million pre-communication answers from 15 open-weight models on 529 Fermi questions, which ask for approximate numerical quantities. They separated each model’s recurring error on a question from variation between repeated answers.
Teams of 30 cut the mean absolute log-scale error, capped at five orders of magnitude, by a median 1.7% versus one answer. A simulated reference that also removes each question’s bias cuts it by 74–79%, supporting the conclusion that shared question errors limit aggregation.
The findings cover open-weight models from 3B to 72B parameters, the prompts and decoding settings tested, and the evaluated Fermi-question sets.
Revision rounds used a parser that sometimes showed agents a different peer estimate from the one recorded, limiting what those rounds can say about debate.
What the evidence shows
- What the team argument assumes
This empirical analysis tests the common argument that combining more answers should reduce error when answers are unbiased and independent. It studies Fermi estimation, where a question asks for an approximate numerical quantity and answers can be combined on a logarithmic scale. Its error score, MAE@5, is the mean absolute log-scale error capped at five orders of magnitude.
- Errors tied to each question
The researchers separate each model’s recurring error on a question from the variation between its repeated answers. In realFP, a set of Fermi estimation questions, the question-level component accounts for a median 0.906 of error variance across questions and 15 models.
That points to errors tied to which question is being answered, rather than showing residual dependence between teammates on a fixed question.
- Two different kinds of alignment
The near-zero teammate residual correlations suggest that answers to a fixed question vary almost independently. Yet different models’ errors across questions remain aligned, suggesting they often miss the same questions in the same direction; the correlations do not establish why.
- Observed gains and a reference
On realFP, teams of 30 reduce error by a median 1.7% compared with one answer. Under the same score, a reference with independent answers unbiased on each question reduces error by 74–79%, supporting the interpretation that shared question errors leave little for real teams to cancel.
- Agreement does not settle accuracy
For five-agent teams, debate sharply reduces within-team spread, but the estimated question-level component stays essentially unchanged. In the second realFP corpus, teams revising after seeing peers were not detectably more accurate than teams revising from their own earlier answers.
Together, these findings distinguish greater agreement from a clear accuracy gain due specifically to peer exchange.
- A bounded composition gain
A five-model mixture of similarly accurate models reduces error by 6.6% compared with those models’ own homogeneous teams, but its advantage over the homogeneous team of its most accurate member is unresolved.
Changing team composition can improve on the constituent models’ own teams in this comparison, while the tested mixture does not establish that a mixed team beats its strongest member.
Deep
- Original paper authors
- Carolina Fortuna, Blaz Bertalanic
- Submitted to arXiv
- License
- CC BY 4.0