Stop Marginalizing My Dreams: Model Inversion via Laplace Kernel for Continual Learning

2026-05-12

AI Summary

This work addresses a key limitation in existing data-free continual learning methods, where model inversion typically assumes diagonal covariance and neglects inter-feature correlations, resulting in low-fidelity synthetic samples and poor knowledge retention. To overcome this, the authors propose REMIX, a novel framework that introduces scalable full-covariance modeling into data-free continual learning for the first time. By parameterizing the covariance structure via Laplacian kernels, REMIX efficiently captures feature dependencies while avoiding the computational burden of dense matrix inversion and determinant calculation, achieving linear memory overhead and only logarithmic additional computational cost. Extensive experiments on standard DFCIL benchmarks demonstrate that REMIX significantly outperforms prior approaches, underscoring the critical importance of modeling feature correlations for generating high-quality pseudo-samples and enhancing continual learning performance.

Abstract

Data-free continual learning (DFCIL) relies on model inversion to synthesize pseudo-samples and mitigate catastrophic forgetting. However, existing inversion methods are fundamentally limited by a simplifying assumption: they model feature distributions using diagonal covariance, effectively ignoring correlations that define the geometry of learned representations. As a result, synthesized samples often lack fidelity, limiting knowledge retention. In this work, we show that modeling feature dependencies is a key ingredient for effective DFCIL. We introduce REMIX, a structured covariance modeling framework that enables scalable full-covariance modeling without the prohibitive cost of dense matrix inversion and log-determinant computation. By leveraging a Laplace kernel parameterization, REMIX captures structured feature dependencies using memory that scales linearly with the feature dimensionality, while requiring only an additional logarithmic factor in computation. Modeling these correlations produces more coherent synthetic samples and consistently improves performance across standard DFCIL benchmarks. Our results demonstrate that moving beyond diagonal assumptions is essential for effective and scalable data-free continual learning. Our code is available at https://github. com/pkrukowski1/REMIX-Model-Inversion-via-Laplace-Kernel.

Problem

• continual learning
• model inversion
• covariance modeling
• catastrophic forgetting
• data-free

Innovation

• model inversion
• structured covariance
• Laplace kernel
• data-free continual learning
• catastrophic forgetting