Continual learning with the neural tangent ensemble

2024-08-30

Neural Information Processing Systems

AI Summary

This work addresses catastrophic forgetting in continual learning by proposing a forgetting-free learning framework grounded in the Neural Tangent Kernel (NTK) and Bayesian inference. Methodologically, it models deep networks as Bayesian ensembles of Neural Tangent Experts—each associated with a fixed or adaptive classifier. Theoretically, under the lazy training regime, the network is proven equivalent to a fixed-expert ensemble, and its Bayesian posterior update reduces to scaled-projected stochastic gradient descent. This framework unifies static and dynamic ensemble behaviors for the first time, yielding interpretable and computationally tractable parameter update rules. Empirically, it significantly mitigates forgetting on standard continual learning benchmarks, validating both theoretical soundness and practical efficacy.

Technology Category

• Machine Learning: Bayesian Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

A natural strategy for continual learning is to weigh a Bayesian ensemble of fixed functions. This suggests that if a (single) neural network could be interpreted as an ensemble, one could design effective algorithms that learn without forgetting. To realize this possibility, we observe that a neural network classifier with N parameters can be interpreted as a weighted ensemble of N classifiers, and that in the lazy regime limit these classifiers are fixed throughout learning. We call these classifiers the neural tangent experts and show they output valid probability distributions over the labels. We then derive the likelihood and posterior probability of each expert given past data. Surprisingly, the posterior updates for these experts are equivalent to a scaled and projected form of stochastic gradient descent (SGD) over the network weights. Away from the lazy regime, networks can be seen as ensembles of adaptive experts which improve over time. These results offer a new interpretation of neural networks as Bayesian ensembles of experts, providing a principled framework for understanding and mitigating catastrophic forgetting in continual learning settings.

Problem

• Continual learning without forgetting
• Neural networks as Bayesian ensembles
• Mitigating catastrophic forgetting in learning

Innovation

• Neural tangent ensemble interpretation
• Bayesian framework for continual learning
• Mitigating catastrophic forgetting with SGD