Reverse Probing: Evaluating Knowledge Transfer via Finetuned Task Embeddings for Coreference Resolution

2025-01-31

AI Summary

This study investigates effective knowledge transfer from simple NLP tasks—such as coreference resolution, named entity recognition (NER), and relation extraction—to the more complex task of end-to-end coreference resolution. We propose a novel “reverse probing” paradigm: the backbone of a coreference resolution model is frozen, while fine-tuned representations from auxiliary tasks are injected as learnable probes into intermediate encoder layers. Our experiments reveal three key findings: (1) representations from intermediate layers yield significantly stronger transfer performance than those from top layers; (2) semantically proximal tasks—e.g., paraphrase detection—contribute most to performance gains; and (3) integrating multi-task embeddings via an attention-based fusion mechanism improves F₁ by 1.8% on OntoNotes, with robustness confirmed through systematic ablation studies. This work establishes a new perspective on cross-task knowledge transfer and provides a reusable methodological framework for representation injection and fusion in hierarchical NLP models.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

In this work, we reimagine classical probing to evaluate knowledge transfer from simple source to more complex target tasks. Instead of probing frozen representations from a complex source task on diverse simple target probing tasks (as usually done in probing), we explore the effectiveness of embeddings from multiple simple source tasks on a single target task. We select coreference resolution, a linguistically complex problem requiring contextual understanding, as focus target task, and test the usefulness of embeddings from comparably simpler tasks tasks such as paraphrase detection, named entity recognition, and relation extraction. Through systematic experiments, we evaluate the impact of individual and combined task embeddings. Our findings reveal that task embeddings vary significantly in utility for coreference resolution, with semantic similarity tasks (e.g., paraphrase detection) proving most beneficial. Additionally, representations from intermediate layers of fine-tuned models often outperform those from final layers. Combining embeddings from multiple tasks consistently improves performance, with attention-based aggregation yielding substantial gains. These insights shed light on relationships between task-specific representations and their adaptability to complex downstream tasks, encouraging further exploration of embedding-level task transfer.

Problem

• knowledge transfer
• coreference resolution
• text understanding

Innovation

• Reverse Probing
• Attention Mechanism
• Cross-task Synergy