PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection

2025-02-17

AI Summary

Visual instruction tuning suffers from excessive computational overhead due to multimodal data redundancy; existing selection methods—relying on surrogate models or loss-based metrics—require model inference and backpropagation, limiting scalability. This paper introduces the first training-free, surrogate-model-free, and gradient-free multimodal data selection framework. It quantifies data value in a task-adaptive manner by modeling intrinsic modality alignment between visual encodings and language instructions via Pearson correlation analysis, enabling zero-cost data valuation and self-pruning. The method integrates gradient-free scoring with structural analysis of multimodal feature spaces. Evaluated on eight vision-language and three language-only benchmarks, it surpasses full fine-tuning, achieving an average performance gain of 101.7% and reducing end-to-end fine-tuning time to 30% of conventional approaches.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Visual instruction tuning refines pre-trained Multimodal Large Language Models (MLLMs) to enhance their real-world task performance. However, the rapid expansion of visual instruction datasets introduces significant data redundancy, leading to excessive computational costs. Existing data selection methods predominantly rely on proxy models or loss-based metrics, both of which impose substantial computational overheads due to the necessity of model inference and backpropagation. To address this challenge, we propose PRISM, a novel training-free approach for efficient multimodal data selection. Unlike existing methods, PRISM eliminates the reliance on proxy models, warm-up pretraining, and gradient-based optimization. Instead, it leverages Pearson correlation analysis to quantify the intrinsic visual encoding properties of MLLMs, computing a task-specific correlation score to identify high-value instances. This not only enbles data-efficient selection,but maintains the original performance. Empirical evaluations across multiple MLLMs demonstrate that PRISM reduces the overall time required for visual instruction tuning and data selection to just 30% of conventional methods, while surpassing fully fine-tuned models across eight multimodal and three language understanding benchmarks, achieving a 101.7% relative improvement in final performance.

Problem

• Efficient multimodal data selection
• Reduces computational costs
• Improves model performance

Innovation

• Training-free multimodal data selection
• Pearson correlation analysis utilization
• Eliminates proxy models and gradient optimization