OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning

2026-05-28

AI Summary

This work addresses the high computational and memory costs incurred by long visual token sequences in vision-language model (VLM) reasoning, where existing fixed-budget token pruning methods struggle to balance input redundancy and query relevance. The authors propose OccamToken, a training-free dynamic token pruning framework that abandons conventional absolute importance scoring. Instead, it introduces register tokens as stable references and leverages a register attention mechanism coupled with relative evidence testing to achieve image-adaptive redundancy pruning and query-adaptive relevance pruning. Supporting dynamic budgets, OccamToken retains only about 40 tokens (1.4% of the original sequence) on mainstream VLMs such as LLaVA-NeXT while preserving over 93% of the original accuracy, substantially improving the trade-off between efficiency and performance.

Abstract

Vision-language models (VLMs) rely on long visual token sequences for visual understanding, making the prefill stage expensive in both computation and memory. Most existing pruning methods follow an absolute-ranking paradigm, assigning importance scores to visual tokens and retaining a fixed top-K subset. In this work, we argue that this paradigm is fundamentally brittle: attention sinks distort token importance rankings, while image redundancy and query-dependent visual evidence make fixed token budgets unreliable across inputs. We propose OccamToken, a training-free framework that replaces absolute token ranking with register-anchored relative evidence testing. Instead of asking which tokens are globally important, OccamToken evaluates whether a visual token provides information beyond a register-based reference. Our key insight is that register tokens naturally absorb low-information attention patterns, making them a stable reference for identifying genuinely informative visual evidence. Based on this principle, OccamToken performs both image-adaptive redundancy pruning and query-adaptive relevance pruning through dynamic thresholds derived from register attention. Across LLaVA-NeXT, LLaVA-v1.5, and Qwen3-VL, OccamToken consistently improves the accuracy-efficiency trade-off without additional training. Notably, on LLaVA-NeXT, it reduces 2,880 visual tokens to approximately 40 while preserving over 93% of the original accuracy, enabling stable visual token compression even in the extreme 1.4% retention regime.

Problem

• vision-language models
• token pruning
• inference efficiency
• visual redundancy
• query-dependent relevance

Innovation

• token pruning
• vision-language models
• training-free
• adaptive thresholding
• register-anchored