Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector

2025-02-08

AI Summary

This work addresses catastrophic forgetting in Faster R-CNN for two-stage incremental object detection, revealing— for the first time—that forgetting predominantly occurs in the RoI Head classifier, not the regressor. To mitigate this, we propose NSGP-RePRE: (1) Region Prototype Replay (RePRE), which jointly preserves coarse-grained class centroids and fine-grained intra-class variations to alleviate classifier forgetting; and (2) Null-Space Gradient Projection (NSGP), which constrains classifier update gradients to lie within the null space of the feature extractor’s Jacobian, thereby preserving feature extractor parameter alignment. Grounded in architectural decoupling analysis and dynamic incremental alignment design, our framework significantly enhances model stability. Evaluated on Pascal VOC and MS COCO, NSGP-RePRE achieves state-of-the-art performance, substantially reduces classifier forgetting rates, and simultaneously improves both accuracy and robustness across multi-stage incremental detection scenarios.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Neuro-Symbolic Learning
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Catastrophic forgetting is a critical chanllenge for incremental object detection (IOD). Most existing methods treat the detector monolithically, relying on instance replay or knowledge distillation without analyzing component-specific forgetting. Through dissection of Faster R-CNN, we reveal a key insight: Catastrophic forgetting is predominantly localized to the RoI Head classifier, while regressors retain robustness across incremental stages. This finding challenges conventional assumptions, motivating us to develop a framework termed NSGP-RePRE. Regional Prototype Replay (RePRE) mitigates classifier forgetting via replay of two types of prototypes: coarse prototypes represent class-wise semantic centers of RoI features, while fine-grained prototypes model intra-class variations. Null Space Gradient Projection (NSGP) is further introduced to eliminate prototype-feature misalignment by updating the feature extractor in directions orthogonal to subspace of old inputs via gradient projection, aligning RePRE with incremental learning dynamics. Our simple yet effective design allows NSGP-RePRE to achieve state-of-the-art performance on the Pascal VOC and MS COCO datasets under various settings. Our work not only advances IOD methodology but also provide pivotal insights for catastrophic forgetting mitigation in IOD. Code will be available soon.

Problem

• Address catastrophic forgetting in incremental object detection
• Focus on RoI Head classifier forgetting in Faster R-CNN
• Develop NSGP-RePRE framework for state-of-the-art IOD performance

Innovation

• RePRE mitigates classifier forgetting
• NSGP aligns RePRE with learning dynamics
• NSGP-RePRE achieves state-of-the-art performance