Convergence of Continual Learning in Homogeneous Deep Networks

2026-06-29

AI Summary

This work investigates the convergence properties of weakly regularized continual learning in homogeneous deep networks, revealing why global convergence typically fails in sequential multi-task learning. By modeling the task sequence as successive non-convex projections onto sets defined by task intervals and leveraging the homogeneity of the network along with generalization error analysis, the authors establish a unified theoretical framework encompassing both classification and regression. Moving beyond prior limitations to single-task settings or linear models, they rigorously demonstrate that global convergence generally does not hold, yet identify key regularity conditions that guarantee local linear convergence. Convergence guarantees are further established under both random and cyclic task sequences.

Abstract

We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets. This result generalizes prior analyses restricted to either stationary (single-task) deep models or continual linear models. We show that global convergence generally fails, even for simple models linear in data but nonlinear in parameters. Nevertheless, by leveraging results from nonconvex projection theory, we identify regularity properties of homogeneous deep networks that guarantee local linear convergence under random and cyclic task sequences. Finally, we extend our analysis to continual regression, unifying the framework for homogeneous models.

Problem

• continual learning
• homogeneous deep networks
• convergence
• task sequences
• nonconvex projection

Innovation

• continual learning
• homogeneous deep networks
• sequential projections
• nonconvex projection theory
• local linear convergence