SAMA: Semantic Anchor-aligned Augmentation for Unified Low-Resource Multimodal Information Extraction

2026-06-17

AI Summary

This work addresses the challenges of data scarcity in multimodal information extraction tasks—including named entity recognition, relation extraction, and event extraction—where existing data augmentation methods suffer from coarse cross-modal alignment and task fragmentation. The authors propose SAMA, a unified data augmentation framework that introduces structured semantic anchors for the first time to coordinate multiple expert multimodal large language models in generating text. Concurrently, it synthesizes images through an anchor-preserving diffusion mechanism and employs a dual-constraint filtering module to automatically select high-quality samples. SAMA enables unified cross-task modeling with fine-grained cross-modal alignment, consistently outperforming state-of-the-art methods on MNER, MRE, and MEE benchmarks under both fully supervised and low-resource settings, demonstrating strong generalization and robustness.

Abstract

Multimodal Information Extraction (MIE)-covering tasks such as Multimodal Named Entity Recognition (MNER), Relation Extraction (MRE), and Event Extraction (MEE)-is essential for understanding multimedia content but remains constrained by severe data scarcity. Although data augmentation is a promising remedy, existing approaches are impeded by coarse cross-modal alignment and fragmented, task-specific designs that fail to exploit shared semantic knowledge. To overcome these limitations, we introduce Semantic Anchor-aligned Multimodal Augmentation (SAMA), a unified framework for generating high-fidelity, task-aware synthetic data. SAMA constructs structured semantic anchors from ground-truth labels to guide a Collaborative Multi-Experts Multimodal Large Language Model (CME-MLLM), which integrates a Universal Adapter for shared semantics with Task-Specific Adapters to produce diverse yet constraint-compliant textual samples. For image synthesis, SAMA employs an Anchor-Preserving Diffusion mechanism that uses anchor-weighted prompts and latent conditioning to maintain critical semantic anchors while diversifying visual contexts. To eliminate the need for manual verification, SAMA further introduces a Dual-Constraint Filtering module that selects synthetic samples based on both cross-modal consistency and anchor fidelity. Extensive experiments across benchmark datasets for MNER, MRE, and MEE demonstrate that SAMA consistently outperforms state-of-the-art augmentation baselines under both fully supervised and low-resource settings, underscoring its versatility, robustness, and effectiveness.

Problem

• Multimodal Information Extraction
• Data Scarcity
• Cross-modal Alignment
• Low-Resource Learning
• Semantic Knowledge

Innovation

• Semantic Anchors
• Multimodal Data Augmentation
• Anchor-Preserving Diffusion
• Task-Aware Synthesis
• Cross-Modal Consistency