RVL-CDIP scores reflect more than document recognition
Data Quality in RVL-CDIP: A Systematic Audit · arXiv:2610.32126
In plain words
A document-classification test set is supposed to show whether a model can recognize document types. Larson and Leach audited RVL-CDIP, a widely used collection of scanned documents, because its high scores might also reflect flaws in the collection.
They found mislabeled pages, repeated pages across training and testing, and clues such as printed identification codes that predict a page’s category. After correcting several problems together, 13 retrained models scored slightly higher overall.
The audit shows that RVL-CDIP scores mix classification ability with features of this particular collection; it does not settle performance on all documents.
The research
When scores on a document-classification test set get very high, do they show that models can recognize document types, or that they have learned quirks of that particular collection?
RVL-CDIP came from a legal-discovery archive and had no published category criteria, making that distinction important to interpreting its reported accuracy.
Larson and Leach audit RVL-CDIP’s labels, overlap, shortcuts, coverage, and text-reading protocol to show how its defects shape classification scores.
Larson and Leach ask whether high accuracy on RVL-CDIP reflects document classification or a model’s fit to the benchmark’s own properties. Their systematic audit checks labels, overlap between training and test pages, visual shortcuts, coverage, and the text-reading protocol.
They find errors and shortcuts that can affect scores, then retrain models with label corrections, identification-code removal, and detected duplicates addressed together. Across 13 models, accuracy is 0.9 to 2.9 points higher than in the original condition.
This supports their view that RVL-CDIP scores combine classification ability with learning the benchmark’s defects.
The audit covers RVL-CDIP and the models and correction procedures tested. Label errors were judged against criteria written by the authors; the audit did not measure how much pretraining on a source archive containing RVL-CDIP test pages contributed to some models’ results.
What the evidence shows
- What does the score measure?
Larson and Leach’s paper is a systematic audit of RVL-CDIP, a test set for classifying document types. The collection was assembled for legal discovery rather than machine-learning evaluation, and its release did not define what each category should contain.
The audit asks whether high scores reflect recognizing document types or learning properties of this particular collection.
- Labels were not always reliable
The authors wrote category criteria, then a team manually reviewed all 400,000 documents against them. They found that 10.51% had an unknown, wrong, or mixed label; counting uncertain cases as errors raised the rate to 11.48%.
Because another team might draw some category boundaries differently, these rates describe errors under the authors’ published criteria.
- Several shortcuts affect accuracy
Other checks identified different sources of score distortion. A refined search found a duplicate or shared-template training counterpart for 35.19% of test pages, a lower bound.
A classifier using only litigation identification-code features reached 69.2% accuracy across 16 categories. Separately, changing the text-reading software while holding model architecture and training settings constant raised accuracy by 4.1 to 8.3 percentage points across four tested models.
- Joint corrections changed scores
The authors retrained 13 models after correcting labels, removing identification codes, and excluding test pages with a detected training counterpart. Their accuracy was 0.9 to 2.9 percentage points higher than in the original condition.
Since the corrections were combined and the scored test pages differed, this comparison shows the result of joint remediation, not the separate effect of each change.
- Scores did not fully transfer
The authors also tested the same image models on newly collected documents from other sources. Seven image models trained on original labels scored 21.6 to 27.9 percentage points lower there than on the benchmark test split.
The gap persisted for image models trained with corrected labels, while several text and layout models’ gaps nearly disappeared. This comparison covers a small collection with 12 categories, not document classification generally.
Deep
- Original paper authors
- Stefan Larson, Kevin Leach
- Submitted to arXiv
- License
- CC BY 4.0