SAVER: Selective As-Needed Vision Evidence for Multimodal Information Extraction

2026-05-20

AI Summary

This work addresses the challenges of multimodal information extraction from social media, where weakly correlated, redundant, or misleading image-text pairs often render conventional fusion methods computationally inefficient and susceptible to spurious visual cues. To this end, the authors propose SAVER, a novel framework that introduces, for the first time, a confidence-calibrated, on-demand visual activation mechanism. SAVER employs a Conformal Groundability Gate to determine whether visual evidence should be activated, and leverages Clopper–Pearson calibration combined with submodular optimization to select a small, highly relevant, and diverse subset of images. Multimodal representations are aggregated via a Set Transformer, and inference is performed through an energy-based joint scoring head. Evaluated on named entity recognition and relation extraction tasks, SAVER achieves superior F1 performance over strong baselines while significantly reducing AURC, improving activation coverage under risk control, and lowering both FLOPs and P90 latency.

Abstract

Multimodal IE in social media is difficult because a post may attach multiple images that are weakly related, redundant, or even misleading with respect to the text. In this setting, always-on multimodal fusion wastes computation and can amplify spurious visual cues. The core challenge is to decide, for each candidate span or marked entity pair, whether vision should be consulted at all and, if so, which small subset of images provides trustworthy evidence. We propose SAVER, a selective vision-as-needed framework for multimodal named entity recognition and multimodal relation extraction. SAVER uses a Conformal Groundability Gate (CGG) to estimate span-level visual groundability in MNER, derive pair-level activation in MRE from the two marked entities, and calibrate the activation threshold on a held-out split via a conformal-style procedure with Clopper--Pearson upper bounds. When activated, a submodular relevance--diversity selector chooses a compact evidence subset across images, which is then aggregated by a Set Transformer. An energy-inspired joint scoring head combines text, optional visual evidence, text--image consistency, and sparse routing for entity typing or relation classification. Experiments show that SAVER consistently improves F1 over strong text-only and always-on multimodal baselines, while reducing AURC, increasing activation coverage at a fixed risk level, and lowering FLOPs and P90 latency.

Problem

• multimodal information extraction
• visual grounding
• selective fusion
• social media
• evidence selection

Innovation

• Selective Multimodal Fusion
• Conformal Groundability Gate
• Submodular Evidence Selection
• Vision-as-Needed
• Multimodal Information Extraction