Batching changes many language-model grading decisions
Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items · arXiv:2610.40241
In plain words
Language models can grade several pieces of text in one prompt, which saves repeated instructions. But when the model grades one item, it has already read the other items and may have written verdicts for them.
The authors tested five models on harm judgments and short-answer grading. Changing which items shared a prompt changed 11–22% of decisions, even though scores against human labels moved little. The finding applies to these tasks and models; star ratings showed smaller changes.
The research
When language models-grade many items in one prompt, does each item still get the same verdict it would receive alone? The assumption matters because batching saves repeated prompt costs, yet the model reads the other items and its earlier verdicts before grading later ones.
The authors compare single-item grading with batched prompts to measure how changing neighboring items affects language-model verdicts.
The authors compare verdicts for the same items graded alone and in different random batches, alongside simpler prompt changes.
In harm judgments and short-answer grading, new groupings changed 11–22% of decisions across five models, while AUROC against human labels stayed close to the single-item result.
That gap shows why a ranking score alone can miss instability in individual verdicts. Tests that kept batches intact while changing order, and edits to earlier verdicts or neighboring text, further indicate that both batch membership and what the model reads or writes can affect a decision.
The evidence comes from three English text-grading tasks, open-weight models, and a limited set of prompt and decoding conditions. The authors did not meet their pre-set generality criterion on star ratings, where fewer decisions changed.
What the evidence shows
- The batching assumption
The paper examines in-context batching, a way to have a language model-grade multiple items in one prompt. This can reduce repeated prompt costs, but it assumes each verdict depends only on its own item.
Since the model reads the other items and may have already produced their verdicts, the authors ask whether changing a batch changes decisions.
- Changing the item groups
To test that assumption, the authors graded the same sampled items alone and in random batches of 25. They compared decisions across new random partitions with changes caused by renamed item keys or reworded instructions.
For harm judgments and short-answer grading, every one of the five main models changed 11–22% of decisions between partitions, above the largest of those batched baselines.
- Why overall scores miss changes
A score against human labels could make the changes look small: across ten model–task pairs, AUROC changed by only −0.033 to +0.010 between batched and single-item grading. AUROC measures how well the model’s written grades rank items against human labels.
It does not tell whether each item receives the same verdict when its batch changes, so the near-stable score does not contradict the decision changes.
- Tracing context effects
The authors separated the effect of which items share a prompt from changes in order and position by comparing independent batches with nine orderings of the same batches. Membership accounted for 0.11–0.22 of the measured latent dissimilarity across the ten harm and grading pairs.
Editing generations also showed that both neighboring texts and earlier model verdicts can affect a target decision.
Deep
- Original paper authors
- Peng Xie, Amr Alanwar
- Submitted to arXiv
- License
- arXiv non-exclusive