Continual Learning in Modern Hopfield Networks with an Application to Diffusion Models

2026-05-27

AI Summary

This work addresses catastrophic forgetting in diffusion models under continual learning scenarios, where task switches lead to the loss of previously acquired knowledge. The authors propose an analysis and replay mechanism grounded in the energy function of modern Hopfield networks. Introducing the notion of “intrinsic forgetting,” they demonstrate that high-energy (i.e., isolated) samples are more susceptible to forgetting and, for the first time, employ Hopfield energy to quantify forgetting severity. Building on this insight, they design an energy-driven replay strategy. Experiments across MHN, DDPM, and Stable Diffusion models confirm that Hopfield energy effectively tracks forgetting dynamics, and that replaying samples selected based on their energy significantly mitigates forgetting of high-energy instances, thereby enhancing continual learning performance.

Abstract

Generative models, including diffusion models, are increasingly used as foundation models and adapted through sequential fine-tuning, making continual learning an essential problem setting. However, continual learning in such generative models remains poorly understood: after a task change, what aspects of the learned distribution are most easily lost, and what replay samples should be prioritized? We address these questions through the modern Hopfield energy. Recent links between modern Hopfield networks (MHNs) and diffusion models allow analyses in MHNs to be transferred to diffusion models. We introduce intrinsic forgetting as an increase in Hopfield energy after the task change. In tractable settings in an MHN, we prove that high-energy, outlier-like samples undergo a larger energy increase than cluster-like samples, implying that samples located in sharp, isolated basins are more forgettable. We further analyze memory replay and show that replay is particularly effective for high-energy samples, enabling an energy-based selection of replay samples. We validate these predictions in experiments on MHNs and two diffusion models under continual-learning settings: Stable Diffusion and a pixel-space DDPM. In these diffusion models, Hopfield energy tracks reconstruction-based forgetting, and replay experiments reveal energy-dependent mitigation of forgetting that is consistent with the MHN analysis.

Problem

• continual learning
• diffusion models
• forgetting
• memory replay
• Hopfield energy

Innovation

• Modern Hopfield Networks
• Continual Learning
• Diffusion Models
• Intrinsic Forgetting
• Energy-based Replay