Revising RVL-CDIP: Quantifying Errors and Test-Train Overlap

2026-06-30

AI Summary

This study addresses critical data quality issues in the widely used RVL-CDIP document classification dataset, revealing approximately 12% label errors and 35% overlap between training and test sets, which severely compromise the reliability of model evaluation. The work presents the first systematic quantification of these problems and constructs multiple cleaned variants through rigorous data curation, including duplicate removal and label correction. Among these, RVL-CDIP-N is proposed as a new benchmark for out-of-distribution generalization. Experimental results demonstrate that correcting label errors significantly improves model accuracy—by 8.1% on average and up to 14% in some cases—whereas removing duplicates unexpectedly degrades performance, highlighting the nuanced and divergent impacts of different data quality dimensions on model evaluation and generalization.

Abstract

RVL-CDIP is a popular dataset for benchmarking document classifiers. However, the dataset contains ample amounts of label errors as well as non-trivial amounts of test-train overlap, both of which may impact model performance metrics. In this paper, we address these two problems by (1) finding and fixing label errors, and (2) detecting and addressing test-train overlap. We produce several variations of RVL-CDIP with label error and test-train overlap fixes, and benchmark document classification performance on these new RVL-CDIP variations. Our rigorous analysis of RVL-CDIP finds that the corpus contains 12\% label error and approximately 35% test-train duplication. Remediation sees improvements in classification accuracy when errors are removed, but sees decreases in accuracy when duplicates are removed. We additionally evaluate models on RVL-CDIP-N, an out-of-distribution benchmark, finding that training on error-corrected data substantially improves OOD generalization, with supervised models gaining an average of 8.1 percentage points in accuracy and improvements as large as 14 percentage points.

Problem

• label errors
• test-train overlap
• document classification
• dataset bias
• RVL-CDIP

Innovation

• label error correction
• test-train overlap
• document classification
• out-of-distribution generalization
• dataset curation