Rethinking Token Reduction for Large Vision-Language Models

2026-03-23

AI Summary

This work addresses the high computational cost of large vision-language models (LVLMs) in multi-turn visual question answering (MT-VQA), primarily caused by redundant visual tokens and exacerbated by the challenge of preserving information relevant to future, unknown questions. To overcome the limitations of existing heuristic compression methods, the authors propose MetaCompress—a novel framework that unifies token pruning and merging into a learnable, prompt-agnostic compression mapping. They further introduce an efficient training paradigm tailored to optimize visual representations for multi-turn dialogue scenarios. MetaCompress surpasses current approaches across multiple LVLM architectures and MT-VQA benchmarks, achieving a superior trade-off between efficiency and accuracy while maintaining strong generalization across conversation turns.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Vision-Language Models (LVLMs) excel in visual understanding and reasoning, but the excessive visual tokens lead to high inference costs. Although recent token reduction methods mitigate this issue, they mainly target single-turn Visual Question Answering (VQA), leaving the more practical multi-turn VQA (MT-VQA) scenario largely unexplored. MT-VQA introduces additional challenges, as subsequent questions are unknown beforehand and may refer to arbitrary image regions, making existing reduction strategies ineffective. Specifically, current approaches fall into two categories: prompt-dependent methods, which bias toward the initial text prompt and discard information useful for subsequent turns; prompt-agnostic ones, which, though technically applicable to multi-turn settings, rely on heuristic reduction metrics such as attention scores, leading to suboptimal performance. In this paper, we propose a learning-based prompt-agnostic method, termed MetaCompress, overcoming the limitations of heuristic designs. We begin by formulating token reduction as a learnable compression mapping, unifying existing formats such as pruning and merging into a single learning objective. Upon this formulation, we introduce a data-efficient training paradigm capable of learning optimal compression mappings with limited computational costs. Extensive experiments on MT-VQA benchmarks and across multiple LVLM architectures demonstrate that MetaCompress achieves superior efficiency-accuracy trade-offs while maintaining strong generalization across dialogue turns. Our code is available at https://github.com/MArSha1147/MetaCompress.

Problem

• token reduction
• multi-turn VQA
• vision-language models
• visual tokens
• inference efficiency

Innovation

• token reduction
• multi-turn VQA
• learning-based compression
• prompt-agnostic
• vision-language models