Adaptive Prompting for Continual Relation Extraction: A Within-Task Variance Perspective

2024-12-11

AI Summary

To address catastrophic forgetting in continual relation extraction (CRE), this paper proposes a memory-replay-free adaptive prompting method. The approach constructs task-specific prompt pools to model intra-task variation, integrates Prefix-tuning with the Mixture-of-Experts (MoE) paradigm, and incorporates generative knowledge distillation to implicitly consolidate prior knowledge—eliminating the need for explicit storage of historical data. Notably, it is the first work to design prompt structures explicitly from the perspective of intra-task variation, thereby jointly modeling both inter-task discrepancies and intra-task variation. Evaluated on multiple CRE benchmarks, the method consistently outperforms existing prompt-based and replay-free approaches, achieving average F1-score improvements of 3.2–5.7 percentage points.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Prompt Engineering / Prompting
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

To address catastrophic forgetting in Continual Relation Extraction (CRE), many current approaches rely on memory buffers to rehearse previously learned knowledge while acquiring new tasks. Recently, prompt-based methods have emerged as potent alternatives to rehearsal-based strategies, demonstrating strong empirical performance. However, upon analyzing existing prompt-based approaches for CRE, we identified several critical limitations, such as inaccurate prompt selection, inadequate mechanisms for mitigating forgetting in shared parameters, and suboptimal handling of cross-task and within-task variances. To overcome these challenges, we draw inspiration from the relationship between prefix-tuning and mixture of experts, proposing a novel approach that employs a prompt pool for each task, capturing variations within each task while enhancing cross-task variances. Furthermore, we incorporate a generative model to consolidate prior knowledge within shared parameters, eliminating the need for explicit data storage. Extensive experiments validate the efficacy of our approach, demonstrating superior performance over state-of-the-art prompt-based and rehearsal-free methods in continual relation extraction.

Problem

• Catastrophic Forgetting
• Continuous Learning
• Relations Extraction

Innovation

• Adaptive Prompt Pool
• Generative Model
• Catastrophic Forgetting Avoidance