CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

2026-04-14

AI Summary

This work addresses the high computational cost of multimodal large language models (MLLMs) caused by redundant visual tokens and the limited robustness of existing static pruning methods under diverse instructions. The authors propose CLASP, a novel framework that integrates category-adaptive multi-layer Vision Transformer feature fusion with a two-stage dynamic token pruning strategy. First, pivot tokens are selected based on attention saliency; then, a redundancy-aware mechanism identifies completion tokens to be pruned, with the token budget dynamically allocated according to the input prompt. Extensive experiments demonstrate that CLASP consistently outperforms state-of-the-art methods across multiple benchmarks, pruning ratios, and MLLM architectures, achieving substantial visual token compression while maintaining or even enhancing model performance—thus enabling efficient, robust, and instruction-adaptive visual token reduction.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Multimodal Large Language Models (MLLMs) suffer from substantial computational overhead due to the high redundancy in visual token sequences. Existing approaches typically address this issue using single-layer Vision Transformer (ViT) features and static pruning strategies. However, such fixed configurations are often brittle under diverse instructions. To overcome these limitations, we propose CLASP, a plug-and-play token reduction framework based on class-adaptive layer fusion and dual-stage pruning. Specifically, CLASP first constructs category-specific visual representations through multi-layer vision feature fusion. It then performs dual-stage pruning, allocating the token budget between attention-salient pivot tokens for relevance and redundancy-aware completion tokens for coverage. Through class-adaptive pruning, CLASP enables prompt-conditioned feature fusion and budget allocation, allowing aggressive yet robust visual token reduction. Extensive experiments demonstrate that CLASP consistently outperforms existing methods across a wide range of benchmarks, pruning ratios, and MLLM architectures. Code will be available at https://github.com/Yunkaidang/CLASP.

Problem

• Multimodal Large Language Models
• visual token redundancy
• static pruning
• computational overhead
• instruction diversity

Innovation

• class-adaptive fusion
• dual-stage pruning
• multimodal large language models
• visual token reduction
• prompt-conditioned budget allocation