Pseudo-Labeling Driven Refinement of Benchmark Object Detection Datasets via Analysis of Learning Patterns

2025-06-01

AI Summary

Mainstream object detection benchmarks (e.g., MS-COCO) suffer from pervasive annotation imperfections—including missing, incorrect, and imprecise bounding boxes—severely limiting model performance and generalization. To address this, we propose the first learning-pattern-aware, pseudo-label-driven automatic annotation refinement framework. Our method introduces a novel four-stage pipeline: (1) generating robust candidate boxes via invertible geometric transformations; (2) confidence-aware filtering using IoU-weighted fusion; (3) expert-level category verification; and (4) spatial refinement guided by response activation maps (RAM), all without manual re-annotation. We construct MJ-COCO, a high-quality re-annotated dataset, which yields significant AP and AP_S improvements across four major benchmarks (COCO, LVIS, Objects365, and OID). Notably, MJ-COCO adds over 200K newly annotated small-object instances, substantially enhancing coverage and robustness.

Technology Category

• Computer Vision: Object Detection & Categorization
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Benchmark object detection (OD) datasets play a pivotal role in advancing computer vision applications such as autonomous driving, and surveillance, as well as in training and evaluating deep learning-based state-of-the-art detection models. Among them, MS-COCO has become a standard benchmark due to its diverse object categories and complex scenes. However, despite its wide adoption, MS-COCO suffers from various annotation issues, including missing labels, incorrect class assignments, inaccurate bounding boxes, duplicate labels, and group labeling inconsistencies. These errors not only hinder model training but also degrade the reliability and generalization of OD models. To address these challenges, we propose a comprehensive refinement framework and present MJ-COCO, a newly re-annotated version of MS-COCO. Our approach begins with loss and gradient-based error detection to identify potentially mislabeled or hard-to-learn samples. Next, we apply a four-stage pseudo-labeling refinement process: (1) bounding box generation using invertible transformations, (2) IoU-based duplicate removal and confidence merging, (3) class consistency verification via expert objects recognizer, and (4) spatial adjustment based on object region activation map analysis. This integrated pipeline enables scalable and accurate correction of annotation errors without manual re-labeling. Extensive experiments were conducted across four validation datasets: MS-COCO, Sama COCO, Objects365, and PASCAL VOC. Models trained on MJ-COCO consistently outperformed those trained on MS-COCO, achieving improvements in Average Precision (AP) and APS metrics. MJ-COCO also demonstrated significant gains in annotation coverage: for example, the number of small object annotations increased by more than 200,000 compared to MS-COCO.

Problem

• Addresses annotation errors in MS-COCO dataset
• Improves object detection model reliability and generalization
• Proposes automated pseudo-labeling refinement for dataset correction

Innovation

• Loss and gradient-based error detection
• Four-stage pseudo-labeling refinement process
• Spatial adjustment via activation map analysis