Scalable Analytic Classifiers with Associative Drift Compensation for Class-Incremental Learning of Vision Transformers

2026-01-29

AI Summary

Vision Transformers (ViTs) face significant challenges in class-incremental learning, particularly due to the high computational cost of classifier reconstruction and the difficulty of balancing accuracy with scalability. To address these issues, this work proposes Low-Rank Regularized Gaussian Discriminant Analysis (LR-RGDA), which leverages low-rank decomposition and the Woodbury identity to substantially reduce inference complexity. Furthermore, it introduces Hopfield-based Distribution Compensation (HopDC)—a novel, training-free mechanism grounded in continuous Hopfield networks—that dynamically calibrates historical class statistics via associative memory to mitigate representation drift. Without requiring any additional training, the proposed approach achieves state-of-the-art performance across multiple class-incremental benchmarks, significantly enhancing both the accuracy and scalability of ViTs in large-scale scenarios.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Class-incremental learning (CIL) with Vision Transformers (ViTs) faces a major computational bottleneck during the classifier reconstruction phase, where most existing methods rely on costly iterative stochastic gradient descent (SGD). We observe that analytic Regularized Gaussian Discriminant Analysis (RGDA) provides a Bayes-optimal alternative with accuracy comparable to SGD-based classifiers; however, its quadratic inference complexity limits its use in large-scale CIL scenarios. To overcome this, we propose Low-Rank Factorized RGDA (LR-RGDA), a scalable classifier that combines RGDA's expressivity with the efficiency of linear classifiers. By exploiting the low-rank structure of the covariance via the Woodbury matrix identity, LR-RGDA decomposes the discriminant function into a global affine term refined by a low-rank quadratic perturbation, reducing the inference complexity from $\mathcal{O}(Cd^2)$ to $\mathcal{O}(d^2 + Crd^2)$, where $C$ is the class number, $d$ the feature dimension, and $r \ll d$ the subspace rank. To mitigate representation drift caused by backbone updates, we further introduce Hopfield-based Distribution Compensator (HopDC), a training-free mechanism that uses modern continuous Hopfield Networks to recalibrate historical class statistics through associative memory dynamics on unlabeled anchors, accompanied by a theoretical bound on the estimation error. Extensive experiments on diverse CIL benchmarks demonstrate that our framework achieves state-of-the-art performance, providing a scalable solution for large-scale class-incremental learning with ViTs. Code: https://github.com/raoxuan98-hash/lr_rgda_hopdc.

Problem

• Class-Incremental Learning
• Vision Transformers
• Classifier Reconstruction
• Scalability
• Representation Drift

Innovation

• Low-Rank Factorized RGDA
• Hopfield-based Distribution Compensator
• Class-Incremental Learning
• Vision Transformers
• Associative Memory