Multi-Perspective Evidence Synthesis and Reasoning for Unsupervised Multimodal Entity Linking

2026-04-22

AI Summary

This work addresses the limitations of existing unsupervised multimodal entity linking methods, which overly rely on instance-centric features while neglecting multi-perspective evidence and their intricate interdependencies. To overcome this, the authors propose MSR-MEL, a novel framework that integrates offline multi-view evidence synthesis with online large language model (LLM) reasoning. MSR-MEL innovatively introduces group-level evidence, leverages contextualized graph structures to aggregate neighborhood information, and combines an asymmetric teacher–student graph neural network with LLM-based semantic reasoning. Evaluated on standard benchmarks, MSR-MEL substantially outperforms current unsupervised approaches, achieving significantly higher entity linking accuracy.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Multimodal Entity Linking (MEL) is a fundamental task in data management that maps ambiguous mentions with diverse modalities to the multimodal entities in a knowledge base. However, most existing MEL approaches primarily focus on optimizing instance-centric features and evidence, leaving broader forms of evidence and their intricate interdependencies insufficiently explored. Motivated by the observation that human expert decision-making process relies on multi-perspective judgment, in this work, we propose MSR-MEL, a Multi-perspective Evidence Synthesis and Reasoning framework with Large Language Models (LLMs) for unsupervised MEL. Specifically, we adopt a two-stage framework: (1) Offline Multi-Perspective Evidence Synthesis constructs a comprehensive set of evidence. This includes instance-centric evidence capturing the instance-centric multimodal information of mentions and entities, group-level evidence that aggregates neighborhood information, lexical evidence based on string overlap ratio, and statistical evidence based on simple summary statistics. A core contribution of our framework is the synthesis of group-level evidence, which effectively aggregates vital neighborhood information by graph. We first construct LLM-enhanced contextualized graphs. Subsequently, different modalities are jointly aligned through an asymmetric teacher-student graph neural network. (2) Online Multi-Perspective Evidence Reasoning leverages the power of LLM as a reasoning module to analyze the correlation and semantics of the multi-perspective evidence to induce an effective ranking strategy for accurate entity linking without supervision. Extensive experiments on widely used MEL benchmarks demonstrate that MSR-MEL consistently outperforms state-of-the-art unsupervised methods. The source code of this paper was available at: https://anonymous.4open.science/r/MSR-MEL-C21E/.

Problem

• Multimodal Entity Linking
• Unsupervised Learning
• Evidence Synthesis
• Multi-Perspective Reasoning
• Knowledge Base

Innovation

• Multi-perspective Evidence Synthesis
• Unsupervised Multimodal Entity Linking
• Large Language Models
• Graph Neural Networks
• Evidence Reasoning