Model Inversion with Layer-Specific Modeling and Alignment for Data-Free Continual Learning

2025-10-30

AI Summary

In data-free continual learning (DFCL), model inversion for synthetic sample generation faces two key challenges: semantic drift across tasks and high computational overhead. To address these, we propose Progressive Model Inversion (PMI), a layer-wise inversion framework that leverages hierarchical optimization initialization and semantic guidance from pre-trained models (e.g., CLIP) to enable efficient pseudo-image generation. PMI further introduces class-level Gaussian modeling and contrastive feature alignment to significantly mitigate semantic drift between synthetic and real data distributions. Notably, it achieves the first stable, low-iteration, privacy-preserving data replay in large-model DFCL settings. Extensive experiments demonstrate that PMI attains state-of-the-art performance across diverse continual learning benchmarks, improves inversion efficiency by 3.2×, enhances synthetic image fidelity, and exhibits strong cross-model generalization and practical deployability.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Diffusion Models for Vision

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Continual learning (CL) aims to incrementally train a model on a sequence of tasks while retaining performance on prior ones. However, storing and replaying data is often infeasible due to privacy or security constraints and impractical for arbitrary pre-trained models. Data-free CL seeks to update models without access to previous data. Beyond regularization, we employ model inversion to synthesize data from the trained model, enabling replay without storing samples. Yet, model inversion in predictive models faces two challenges: (1) generating inputs solely from compressed output labels causes drift between synthetic and real data, and replaying such data can erode prior knowledge; (2) inversion is computationally expensive since each step backpropagates through the full model. These issues are amplified in large pre-trained models such as CLIP. To improve efficiency, we propose Per-layer Model Inversion (PMI), inspired by faster convergence in single-layer optimization. PMI provides strong initialization for full-model inversion, substantially reducing iterations. To mitigate feature shift, we model class-wise features via Gaussian distributions and contrastive model, ensuring alignment between synthetic and real features. Combining PMI and feature modeling, our approach enables continual learning of new classes by generating pseudo-images from semantic-aware projected features, achieving strong effectiveness and compatibility across multiple CL settings.

Problem

• Addresses catastrophic forgetting in continual learning without stored data
• Reduces computational cost of model inversion through layer-specific optimization
• Mitigates feature drift between synthetic and real training data

Innovation

• Per-layer inversion accelerates full-model optimization
• Gaussian modeling aligns synthetic and real features
• Generates pseudo-images from semantic-aware projected features