Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learning

2024-08-29

arXiv.org

AI Summary

This work addresses the lack of a unified theoretical framework explaining generalization, knowledge transfer, and forgetting in overparameterized models—such as deep neural networks—under multi-task learning (MTL) and replay-based continual learning (CL). We establish, for the first time, a unified analytical framework for overparameterized linear models in both MTL and replay-based CL. Leveraging high-dimensional random matrix theory, bias-variance decomposition, and generalization bound derivation, we quantitatively characterize how model capacity, data scale, task similarity, and replay buffer size jointly influence generalization error, forward transfer, and forgetting rate. Our theoretical predictions align closely with large-scale deep network experiments: increasing task similarity reduces forgetting by over 50%; optimal buffer size scales with the square root of the number of tasks. The results provide actionable, theoretically grounded guidance for designing models and replay buffers in continual and multi-task learning settings.

Technology Category

• Machine Learning: Learning Theory
• Natural Language Processing: Learning & Optimization for NLP
• Multiagent Systems: Multiagent Learning

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Multi-task learning (MTL) is a machine learning paradigm that aims to improve the generalization performance of a model on multiple related tasks by training it simultaneously on those tasks. Unlike MTL, where the model has instant access to the training data of all tasks, continual learning (CL) involves adapting to new sequentially arriving tasks over time without forgetting the previously acquired knowledge. Despite the wide practical adoption of CL and MTL and extensive literature on both areas, there remains a gap in the theoretical understanding of these methods when used with overparameterized models such as deep neural networks. This paper studies the overparameterized linear models as a proxy for more complex models. We develop theoretical results describing the effect of various system parameters on the model's performance in an MTL setup. Specifically, we study the impact of model size, dataset size, and task similarity on the generalization error and knowledge transfer. Additionally, we present theoretical results to characterize the performance of replay-based CL models. Our results reveal the impact of buffer size and model capacity on the forgetting rate in a CL setup and help shed light on some of the state-of-the-art CL methods. Finally, through extensive empirical evaluations, we demonstrate that our theoretical findings are also applicable to deep neural networks, offering valuable guidance for designing MTL and CL models in practice.

Problem

• Theoretical gap in overparameterized models for multi-task learning
• Impact of system parameters on generalization error and knowledge transfer
• Characterizing performance of replay-based continual learning methods

Innovation

• Overparameterized linear models as theoretical proxy
• Analyzing system parameters impact on MTL performance
• Characterizing replay-based CL with buffer size effects