Plug-and-Adapt: Multimodal Coreference Resolution at First Sight with a Pretrained Alignment Model

2026-06-16

AI Summary

This work addresses the challenges of multimodal coreference resolution, which typically relies on task-specific annotations and struggles with zero-shot deployment, while existing vision-language foundation models are often limited by scale or API constraints, hindering practical applicability. The authors propose the first lightweight, plug-and-play framework that requires no fine-tuning, leveraging a pretrained fine-grained alignment model and integrating visual and categorical cues through Dempster–Shafer evidence theory to aggregate multimodal similarities for zero-shot coreference resolution. Evaluated on the CIN benchmark, the method outperforms current task-specific approaches and mainstream vision-language foundation models by 5.31% and 2.12% in CoNLL F1 score, respectively, while demonstrating strong robustness and cross-domain generalization capabilities.

Abstract

Visual information helps resolve ambiguity in coreference resolution, leading to notable performance gains. However, existing Multi-modal Coreference Resolution (MCR) methods require training with (partially) annotated data from the target dataset before they can be applied, preventing their direct usability and raising concerns about generalization. While Vision-Language Large Models (VLLMs) with billions of parameters offer promising zero-shot capabilities, they remain largely inaccessible. Their massive size limits deployability, and many are only accessible through paid APIs. In this paper, we propose a plug-and-adapt method that strategically adapts a carefully pre-trained \emph{alignment model} for immediate use in MCR tasks, designed to eliminate the need for training on scarce benchmark datasets or relying on resource-intensive VLLMs. Specifically, we first pre-train a fine-grained alignment model between textual and visual contextual information using vision-language alignment datasets. We then repurpose the alignment model to MCR through similarity aggregation by fusing visual and categorical cues with evidence theory, thereby enhancing effectiveness. Experiments on the Coreference Image Narratives (CIN) benchmark dataset demonstrate the effectiveness of our method, achieving a 5.31\% and 2.12\% improvement in CoNLL F1 over SOTA dedicated methods and popular VLLMs, respectively. We further evaluate our method on a masked CIN dataset for robustness testing and on a specially constructed VCR-MCR dataset for generalization assessment, with results confirming both capabilities.

Problem

• Multimodal Coreference Resolution
• Zero-shot Learning
• Vision-Language Alignment
• Model Generalization
• Annotation Efficiency

Innovation

• multimodal coreference resolution
• alignment model
• plug-and-adapt
• zero-shot learning
• evidence theory