Multi-level Collaborative Distillation Meets Global Workspace Model: A Unified Framework for OCIL

2025-08-12

AI Summary

Online Class-Incremental Learning (OCIL) faces a fundamental trade-off between stability and plasticity under severe memory constraints: replay-based methods fail with extremely limited memory, while ensemble approaches compromise stability. To address this, we propose GWS-MCD—a novel framework featuring an implicitly shared Global Workspace (GWS) that serves as a cross-task knowledge anchor, coupled with a multi-level collaborative distillation mechanism. This mechanism integrates student parameter fusion and periodic knowledge feedback to jointly optimize historical knowledge retention and new-task adaptation. Student-wise consistency regularization and GWS-guided knowledge alignment further mitigate catastrophic forgetting. Evaluated on three standard OCIL benchmarks under stringent memory budgets (as low as 10 samples per class), GWS-MCD achieves state-of-the-art performance—marking the first method to simultaneously enhance both stability and plasticity under ultra-low-memory constraints.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Online Class-Incremental Learning (OCIL) enables models to learn continuously from non-i.i.d. data streams and samples of the data streams can be seen only once, making it more suitable for real-world scenarios compared to offline learning. However, OCIL faces two key challenges: maintaining model stability under strict memory constraints and ensuring adaptability to new tasks. Under stricter memory constraints, current replay-based methods are less effective. While ensemble methods improve adaptability (plasticity), they often struggle with stability. To overcome these challenges, we propose a novel approach that enhances ensemble learning through a Global Workspace Model (GWM)-a shared, implicit memory that guides the learning of multiple student models. The GWM is formed by fusing the parameters of all students within each training batch, capturing the historical learning trajectory and serving as a dynamic anchor for knowledge consolidation. This fused model is then redistributed periodically to the students to stabilize learning and promote cross-task consistency. In addition, we introduce a multi-level collaborative distillation mechanism. This approach enforces peer-to-peer consistency among students and preserves historical knowledge by aligning each student with the GWM. As a result, student models remain adaptable to new tasks while maintaining previously learned knowledge, striking a better balance between stability and plasticity. Extensive experiments on three standard OCIL benchmarks show that our method delivers significant performance improvement for several OCIL models across various memory budgets.

Problem

• Enhances ensemble learning via Global Workspace Model for OCIL
• Balances stability and plasticity in non-i.i.d. data streams
• Improves memory efficiency and cross-task consistency in OCIL

Innovation

• Global Workspace Model integrates student parameters
• Multi-level distillation ensures peer-to-peer consistency
• Dynamic anchor stabilizes cross-task learning