Collaborative Multi-LoRA Experts with Achievement-based Multi-Tasks Loss for Unified Multimodal Information Extraction

2025-05-08

AI Summary

To address gradient conflict, sample imbalance, and excessive task coupling in multi-modal information extraction (MIE), this paper proposes a collaborative multi-LoRA expert architecture. It integrates universal cross-task experts with task-specific experts to enable knowledge sharing while decoupling tasks. A dynamic multi-task loss function—guided by task achievement metrics—is introduced to mitigate training asynchrony. The method unifies low-rank adaptation (LoRA), vision adapters, instruction tuning, and achievement-aware learning. Evaluated on seven benchmark datasets across three MIE task categories, our approach significantly outperforms full-parameter fine-tuning and standard LoRA, achieving superior performance with comparable parameter counts. This demonstrates both high parameter efficiency and strong generalization capability.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Information Extraction

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Multimodal Information Extraction (MIE) has gained attention for extracting structured information from multimedia sources. Traditional methods tackle MIE tasks separately, missing opportunities to share knowledge across tasks. Recent approaches unify these tasks into a generation problem using instruction-based T5 models with visual adaptors, optimized through full-parameter fine-tuning. However, this method is computationally intensive, and multi-task fine-tuning often faces gradient conflicts, limiting performance. To address these challenges, we propose collaborative multi-LoRA experts with achievement-based multi-task loss (C-LoRAE) for MIE tasks. C-LoRAE extends the low-rank adaptation (LoRA) method by incorporating a universal expert to learn shared multimodal knowledge from cross-MIE tasks and task-specific experts to learn specialized instructional task features. This configuration enhances the model's generalization ability across multiple tasks while maintaining the independence of various instruction tasks and mitigating gradient conflicts. Additionally, we propose an achievement-based multi-task loss to balance training progress across tasks, addressing the imbalance caused by varying numbers of training samples in MIE tasks. Experimental results on seven benchmark datasets across three key MIE tasks demonstrate that C-LoRAE achieves superior overall performance compared to traditional fine-tuning methods and LoRA methods while utilizing a comparable number of training parameters to LoRA.

Problem

• Unifies multimodal extraction tasks to avoid separate processing
• Reduces computational cost and gradient conflicts in multi-task learning
• Balances training progress across tasks with varying sample sizes

Innovation

• Collaborative multi-LoRA experts for unified MIE tasks
• Achievement-based multi-task loss balances training progress
• Universal and task-specific experts enhance generalization