Voters increasingly rejected both answers from unchanged AI model pairs
AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models · arXiv:2611.14333
In plain words
People sometimes say an AI got worse, but judges may become harder to satisfy even when the model versions stay the same. This study followed blind votes on fixed-version model pairs on Chatbot Arena.
Voters more often said both answers were bad over time. In 2024, among decisive votes on pairs whose Arena strength ratings differed by at least 20 points, the stronger model’s share stayed steady.
Rejection rose while relative preference in this subset did not; the records cannot tell whether voters raised their standards or examined answers more closely.
The research
When human ratings guide model comparisons or decisions about whether a system has regressed, harsher judgments can be mistaken for a change in the system itself.
This study shows that ratings of unchanged model pairs can shift over time, so a rating’s meaning depends partly on when and by whom it was recorded.
The study tracks blind votes on fixed-version model pairs and finds that rejection of both answers rose while the stronger model’s share stayed flat among decisive votes on pairs whose Arena strength ratings differed by at least 20 points in 2024.
The researchers analyzed blind votes on unchanged model pairs from May 2023 to August 2024, separating votes that rejected both answers from decisive choices.
Rejection rose within the same pairs, while the stronger model’s share among decisive votes on pairs whose Arena strength ratings differed by at least 20 points showed no monthly trend in 2024.
This supports a change in how answers were judged, rather than a changing relative ranking in that measure. Comparisons within judge identifiers and checks for measured prompts, model groups and interface periods support that reading.
The voters were a self-selected public, and the log contains no prompt or answer text. It cannot show whether people raised their acceptance threshold or scrutinized answers more closely, or whether exposure to other AI caused the change.
What the evidence shows
- Separate rejection from preference
The paper’s question is whether harsher judgments of AI mean that models worsened or that people’s standards changed. Chatbot Arena lets the researchers examine blind comparisons of fixed model versions over time.
Because voters can reject both answers as well as choose a winner, the study can track acceptability separately from relative preference.
- Rejection rose, ranking stayed flat
On the same frozen model pairs, outright rejection rose over the study period, while a separate 2024 measure of which model won among decisive votes on pairs rated at least 20 Elo points apart showed no trend. This suggests harsher acceptability judgments without a shift in that relative measure.
- The same identifiers grew harsher
A change in who voted could make later results look harsher, so the researchers compared votes from the same judge identifiers over time. In that subset, “both are bad” votes rose by 1.90 percentage points per month.
Since an identifier may refer to a device or network rather than one person, the result tracks identifiers, not confirmed individuals.
- Checks point to shared change
The increase stayed positive across checks for measured prompt features, selected fixed-model groups and periods between recorded interface changes. A further comparison found that month-to-month changes measured on other model pairs largely removed the trend on each model’s own pairs.
Together, these patterns support a shared change in evaluations, while leaving its precise cause unresolved.
Deep
- Original paper authors
- Zihan Chen, Lei Nico Zheng, Di Zhu
- Submitted to arXiv
- License
- CC BY 4.0