Mitigating Task-Order Sensitivity and Forgetting via Hierarchical Second-Order Consolidation

2026-01-31

AI Summary

This work addresses the high performance variance and catastrophic forgetting in continual learning caused by random task ordering. To mitigate these issues, the authors propose the HTCL framework, which integrates multi-scale knowledge through a hierarchical Taylor expansion: enabling rapid local adaptation to new tasks while conservatively consolidating knowledge globally via Hessian-based regularization. HTCL introduces a model-agnostic hierarchical consolidation mechanism that offers theoretical guarantees and robustness to task sequence variations. Extensive experiments across multiple benchmarks demonstrate that HTCL improves average accuracy by 7%–25% and reduces the standard deviation of final accuracy by up to 68%, significantly alleviating catastrophic forgetting and enhancing learning stability.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

We introduce $\textbf{Hierarchical Taylor Series-based Continual Learning (HTCL)}$, a framework that couples fast local adaptation with conservative, second-order global consolidation to address the high variance introduced by random task ordering. To address task-order effects, HTCL identifies the best intra-group task sequence and integrates the resulting local updates through a Hessian-regularized Taylor expansion, yielding a consolidation step with theoretical guarantees. The approach naturally extends to an $L$-level hierarchy, enabling multiscale knowledge integration in a manner not supported by conventional single-level CL systems. Across a wide range of datasets and replay and regularization baselines, HTCL acts as a model-agnostic consolidation layer that consistently enhances performance, yielding mean accuracy gains of $7\%$ to $25\%$ while reducing the standard deviation of final accuracy by up to $68\%$ across random task permutations.

Problem

• task-order sensitivity
• catastrophic forgetting
• continual learning
• performance variance

Innovation

• Hierarchical Continual Learning
• Second-Order Consolidation
• Task-Order Sensitivity
• Hessian-Regularized Taylor Expansion
• Model-Agnostic Consolidation