Prototype Latent World Model Replay for Class-Incremental Learning

2026-06-28

AI Summary

This work addresses class-incremental learning without access to any original samples from previous classes by proposing a memory-free framework. It leverages a frozen ImageNet-pretrained encoder to construct a stable latent space, models the latent distributions of old classes using prototype-centered representations, and jointly optimizes the geometric structure of both old and new classes through lightweight adapters and supervised contrastive learning. To the best of our knowledge, this is the first approach to achieve effective incremental learning without storing any raw images. The method significantly outperforms existing techniques on Split CIFAR-100, achieving LastAcc of 31.64%, 37.06%, and 43.10% and AvgAcc of 45.86%, 52.19%, and 56.18% under the Inc5, Inc10, and Inc20 settings, respectively.

Abstract

Class-incremental learning requires a model to learn new classes while preserving decision regions for old ones. This is difficult when raw old samples are no longer available. We propose Prototype Latent World Model Replay, a memory-free framework that stores old classes as distributions over stable hidden states rather than as images. A frozen ImageNet-pretrained encoder maps each image into a latent state space. In this space, each class is summarized by several prototype-centered distributions with class-specific variances. When new classes arrive, the model samples old latent states from this prototype world model. It then trains a lightweight adapter and classifier using both sampled old states and real new-class features. We also add a supervised contrastive term in the adapter space to promote intra-class compactness and old-new class separation. On Split CIFAR-100, our method improves over fine-tuning under Inc5, Inc10, and Inc20 without storing raw exemplars. The full Ours-LWM+Con model raises LastAcc from 4.55% to 31.64%, from 9.06% to 37.06%, and from 16.96% to 43.10% in Inc5, Inc10, and Inc20, respectively. It also achieves AvgAcc of 45.86%, 52.19%, and 56.18%. Ablation and retention analyses show that stable latent-state replay is the main source of the gain. Contrastive separation further refines the old-new geometry. These results suggest that prototype latent memory preserves reusable class-state distributions, rather than only fitting the current classifier.

Problem

• class-incremental learning
• catastrophic forgetting
• memory-free
• old class preservation
• latent replay

Innovation

• Class-Incremental Learning
• Latent Replay
• Prototype-based Modeling
• Memory-Free Learning
• Supervised Contrastive Learning