Sculpting Margin Penalty: Intra-Task Adapter Merging and Classifier Calibration for Few-Shot Class-Incremental Learning

2025-08-07

AI Summary

In few-shot class-incremental learning, balancing base-class discriminability and novel-class generalizability remains challenging due to data privacy constraints and high annotation costs. To address this, we propose Margin-aware Intra-task Adapter Merging (MIAM), a novel framework with three key contributions: (1) a dual-branch low-rank adapter architecture, trained in stages using margin-penalized and unconstrained classification losses, respectively; (2) Margin-Preserving Classifier Calibration (MPCC), a data-free strategy that refines decision boundaries across all classes without access to original training data; and (3) an adaptive weight fusion mechanism that dynamically integrates outputs from both adapters to enhance forward compatibility and decision boundary clarity. Extensive experiments on CIFAR-100, ImageNet-R, and CUB-200 demonstrate that MIAM achieves superior trade-offs between base- and novel-class accuracy, outperforming prior state-of-the-art methods.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Real-world applications often face data privacy constraints and high acquisition costs, making the assumption of sufficient training data in incremental tasks unrealistic and leading to significant performance degradation in class-incremental learning. Forward-compatible learning, which prospectively prepares for future tasks during base task training, has emerged as a promising solution for Few-Shot Class-Incremental Learning (FSCIL). However, existing methods still struggle to balance base-class discriminability and new-class generalization. Moreover, limited access to original data during incremental tasks often results in ambiguous inter-class decision boundaries. To address these challenges, we propose SMP (Sculpting Margin Penalty), a novel FSCIL method that strategically integrates margin penalties at different stages within the parameter-efficient fine-tuning paradigm. Specifically, we introduce the Margin-aware Intra-task Adapter Merging (MIAM) mechanism for base task learning. MIAM trains two sets of low-rank adapters with distinct classification losses: one with a margin penalty to enhance base-class discriminability, and the other without margin constraints to promote generalization to future new classes. These adapters are then adaptively merged to improve forward compatibility. For incremental tasks, we propose a Margin Penalty-based Classifier Calibration (MPCC) strategy to refine decision boundaries by fine-tuning classifiers on all seen classes' embeddings with a margin penalty. Extensive experiments on CIFAR100, ImageNet-R, and CUB200 demonstrate that SMP achieves state-of-the-art performance in FSCIL while maintaining a better balance between base and new classes.

Problem

• Balancing base-class discriminability and new-class generalization in FSCIL
• Addressing ambiguous inter-class decision boundaries in incremental tasks
• Enhancing forward compatibility with margin penalties in few-shot learning

Innovation

• Margin-aware Intra-task Adapter Merging mechanism
• Margin Penalty-based Classifier Calibration strategy
• Parameter-efficient fine-tuning with margin penalties