ReFineG: Synergizing Small Supervised Models and LLMs for Low-Resource Grounded Multimodal NER

2025-09-13

AI Summary

To address two key challenges in low-resource multimodal named entity recognition (GMNER)—the heavy reliance of supervised models on costly manual annotations and the domain knowledge conflicts inherent in multimodal large language models (MLLMs)—this paper proposes a three-stage collaborative framework. First, domain-aware synthetic data generation alleviates annotation scarcity. Second, an uncertainty-estimation-based task allocation mechanism dynamically partitions responsibilities between textual mention detection and visual grounding subtasks. Third, analogy-driven multimodal context selection mitigates LLM domain-knowledge interference. The framework freezes MLLM parameters and tightly couples a lightweight supervised model with an uncertainty estimation module. Evaluated on the CCKS2025 GMNER benchmark, it achieves an F1 score of 0.6461, ranking second overall, and demonstrates significant improvements in few-shot generalization and cross-modal alignment accuracy.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Grounded Multimodal Named Entity Recognition (GMNER) extends traditional NER by jointly detecting textual mentions and grounding them to visual regions. While existing supervised methods achieve strong performance, they rely on costly multimodal annotations and often underperform in low-resource domains. Multimodal Large Language Models (MLLMs) show strong generalization but suffer from Domain Knowledge Conflict, producing redundant or incorrect mentions for domain-specific entities. To address these challenges, we propose ReFineG, a three-stage collaborative framework that integrates small supervised models with frozen MLLMs for low-resource GMNER. In the Training Stage, a domain-aware NER data synthesis strategy transfers LLM knowledge to small models with supervised training while avoiding domain knowledge conflicts. In the Refinement Stage, an uncertainty-based mechanism retains confident predictions from supervised models and delegates uncertain ones to the MLLM. In the Grounding Stage, a multimodal context selection algorithm enhances visual grounding through analogical reasoning. In the CCKS2025 GMNER Shared Task, ReFineG ranked second with an F1 score of 0.6461 on the online leaderboard, demonstrating its effectiveness with limited annotations.

Problem

• Improving low-resource Grounded Multimodal NER performance
• Addressing domain knowledge conflict in multimodal LLMs
• Reducing reliance on costly multimodal annotations

Innovation

• Three-stage collaborative framework integration
• Domain-aware NER data synthesis strategy
• Uncertainty-based prediction refinement mechanism