AI reviewers catch paper errors, but results depend on setup
Auditing AI peer reviewers: a benchmark on real scientific papers · arXiv:2611.17052
In plain words
This paper introduces a way to test AI paper reviewers by planting specific mistakes in real research papers and checking whether reviewers find them. It also gives each paper a clean copy to see whether reviewers raise false alarms.
The authors’ reviewer, Skepthical, caught a larger share of planted errors than a structured GPT-5.5 reviewer on the astrophysics papers. On the machine-learning papers, their overall detection rates were not significantly different.
The research
A detection score needs to show both what a reviewer catches and how often its flags are false before it can describe its performance. Here, model and prompt choices shift detection substantially, so one score cannot be read as a property of AI reviewers in general.
The paper introduces a test pipeline for AI paper reviewers and Skepthical, a multi-agent reviewer that checks general, mathematical, numerical, and citation claims.
The pipeline planted four kinds of errors at different severity levels in papers from two corpora of 15 preprints each. A cross-family language-model panel judged whether reviewers found them, while clean copies were used to assess false alarms.
Skepthical detected 70.6% of injected errors in astrophysics, versus 61.3% for structured GPT-5.5, a significant difference. In machine learning, the rates were 76.7% and 81.1%, with no significant difference.
The evidence supports a corpus-specific advantage on astrophysics, not a clear overall separation on machine learning.
Each corpus contains 15 papers from one field and one recent day. Skepthical uses more model calls and three providers, so its advantage cannot be attributed to pipeline design alone; the test measures planted defects, not overall review quality, and a human checked detection decisions for only one astrophysics paper.
What the evidence shows
- A test for concrete errors
The evaluation pipeline for AI paper reviewers tests whether systems detect planted errors in real papers and whether they raise false alarms on clean copies. The paper also introduces Skepthical, a multi-agent reviewer with separate checks for general issues, mathematics, numerical claims, and citations.
Together, the test and reviewer make it possible to compare concrete error detection across different error types and severities.
- The advantage varies by corpus
The comparison favors Skepthical on the astrophysics corpus, but shows no clear overall difference on the machine-learning corpus. This result is specific to these tests, and the systems’ different amounts of computation and providers mean it does not isolate the pipeline’s contribution.
- Model choice changes detection
Even with the same free-form prompt, three single-model reviewers differed in their detection rates by as much as 48.6 percentage points on astrophysics and 52.8 points on machine learning. Those gaps compare the highest and lowest rates among the three models on each corpus.
They show that the measured result depends on which model is asked to review under that prompt, rather than describing AI reviewers as one uniform group.
- Instructions also change results
A comparison within Claude Code held the model, sandbox, available tools, and arXiv access fixed while changing the prompt. The structured prompt raised detection by 32.2 percentage points on astrophysics and 38.9 points on machine learning.
This provides evidence that instructions can change results in this setup; it does not explain the full Skepthical comparison or establish an effect for every reviewer.
Deep
- Original paper authors
- Íñigo Zubeldia, Boris Bolliet, Francisco Villaescusa-Navarro, Pablo Villanueva-Domingo, Miles Cranmer
- Submitted to arXiv
- License
- CC BY 4.0