Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learning

2024-08-02

arXiv.org

AI Summary

To address severe catastrophic forgetting and the neglect of textual semantic information in continual learning, this paper introduces, for the first time, semantic knowledge from the CLIP pre-trained text encoder. We propose a dual-module framework comprising Semantic-Guided Representation Learning (SG-RL) and Semantic-Guided Knowledge Distillation (SG-KD). SG-RL models inter-class similarity via cross-task semantic alignment, incorporating soft label assignment and contrastive representation regularization; SG-KD transfers semantic-aware knowledge to preserve learned representations. Together, these modules enable joint optimization of visual representations and semantic knowledge. Evaluated on both generic and fine-grained continual learning benchmarks, our method surpasses state-of-the-art approaches, achieving average accuracy gains of 3.2–5.7 percentage points. It effectively mitigates forgetting while enhancing consistency in discrimination between old and new classes.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Computer Vision: Representation Learning for Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Deep neural networks (DNNs) excel on fixed datasets but struggle with incremental and shifting data in real-world scenarios. Continual learning addresses this challenge by allowing models to learn from new data while retaining previously learned knowledge. Existing methods mainly rely on visual features, often neglecting the rich semantic information encoded in text. The semantic knowledge available in the label information of the images, offers important semantic information that can be related with previously acquired knowledge of semantic classes. Consequently, effectively leveraging this information throughout continual learning is expected to be beneficial. To address this, we propose integrating semantic guidance within and across tasks by capturing semantic similarity using text embeddings. We start from a pre-trained CLIP model, employ the emph{Semantically-guided Representation Learning (SG-RL)} module for a soft-assignment towards all current task classes, and use the Semantically-guided Knowledge Distillation (SG-KD) module for enhanced knowledge transfer. Experimental results demonstrate the superiority of our method on general and fine-grained datasets. Our code can be found in https://github.com/aprilsveryown/semantically-guided-continual-learning.

Problem

• Enhancing continual learning with semantic text embeddings
• Leveraging label semantics for knowledge retention
• Improving knowledge transfer across incremental tasks

Innovation

• Integrate semantic guidance using text embeddings
• Employ Semantically-guided Representation Learning module
• Use Semantically-guided Knowledge Distillation module