DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference

2026-02-21

AI Summary

This work addresses the high computational cost of vision-language models caused by dense visual tokens, a challenge often exacerbated by existing compression methods that compromise accuracy. The authors propose a two-stage unified framework for efficient token compression: first applying redundancy-aware compression at the output of the vision encoder, followed by text-guided, layer-wise pruning of visual tokens within the language backbone. This approach introduces a novel collaborative compression mechanism that jointly optimizes information retention and computational efficiency, supports end-to-end training, and is applicable to both image and video multimodal inputs. Evaluated on LLaVA-1.5-7B, the method achieves an 89% reduction in visual tokens while preserving over 97% of the original accuracy; on Video-LLaVA-7B, it attains a 53.1% compression rate with performance surpassing the baseline, reaching 100% relative accuracy.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision-language models (VLMs) have achieved remarkable multimodal understanding and reasoning capabilities, yet remain computationally expensive due to dense visual tokenization. Existing efficiency approaches either merge redundant visual tokens or drop them progressively in language backbone, often trading accuracy for speed. In this work, we propose DUET-VLM, a versatile plug-and-play dual compression framework that consists of (a) vision-only redundancy aware compression of vision encoder's output into information-preserving tokens, followed by (b) layer-wise, salient text-guided dropping of visual tokens within the language backbone to progressively prune less informative tokens. This coordinated token management enables aggressive compression while retaining critical semantics. On LLaVA-1.5-7B, our approach maintains over 99% of baseline accuracy with 67% fewer tokens, and still retains >97% even at 89% reduction. With this dual-stage compression during training, it achieves 99.7% accuracy at 67% and 97.6% at 89%, surpassing prior SoTA visual token reduction methods across multiple benchmarks. When integrated into Video-LLaVA-7B, it even surpasses the baseline -- achieving >100% accuracy with a substantial 53.1% token reduction and retaining 97.6% accuracy under an extreme 93.4% setting. These results highlight end-to-end training with DUET-VLM, enabling robust adaptation to reduced visual (image/video) input without sacrificing accuracy, producing compact yet semantically rich representations within the same computational budget. Our code is available at https://github.com/AMD-AGI/DUET-VLM.

Problem

• vision-language models
• visual token redundancy
• computational efficiency
• token reduction
• multimodal understanding

Innovation

• token reduction
• vision-language models
• dual-stage compression
• text-guided pruning
• efficient multimodal learning