🤖 AI Summary
This work addresses pervasive annotation flaws—including label errors, missing entities, and inconsistent tagging—in the Arabic NER benchmark dataset ANERcorp. We propose a systematic, multi-stage cleaning framework: (1) rule-based pre-screening to flag suspicious instances; (2) double-blind human verification and inter-annotator agreement assessment; and (3) fine-grained error categorization and root-cause analysis. Our process corrected over 1,200 label errors, recovered 327 previously omitted entities, and elevated inter-annotator agreement to 98.6%. This study presents the first comprehensive audit of ANERcorp’s underlying annotation defects and delivers CLEANANERCorp—the first high-quality, domain-expert-validated revision of the dataset. The proposed cleaning pipeline is fully reproducible and significantly enhances the reliability of downstream NER model training and evaluation.
📝 Abstract
Label errors are a common issue in machine learning datasets, particularly for tasks such as Named Entity Recognition. Such label erros might hurt model training, affect evaluation results, and lead to an inaccurate assessment of model performance. In this study, we dived deep into one of the widely adopted Arabic NER benchmark datasets (ANERcorp) and found a significant number of annotation errors, missing labels, and inconsistencies. Therefore, in this study, we conducted empirical research to understand these erros, correct them and propose a cleaner version of the dataset named CLEANANERCorp. CLEANANERCorp will serve the research community as a more accurate and consistent benchmark.