An AI4AI Framework for Visual Token Pruning
This work addresses the limitations of existing vision token pruning methods, which rely on handcrafted heuristics and struggle to generalize across diverse model architectures and pruning objectives. To overcome this, the authors propose AutoPrune, a training-free AI4AI framework that leverages large language models (LLMs) to automatically design effective pruning strategies. Key innovations include TPDSL, a domain-specific language tailored for token pruning; a residual representation of search states to emphasize critical strategy components; and structured guidance for LLMs to generate algorithms within a constrained space. Extensive experiments demonstrate AutoPrune’s superior performance across 14 benchmarks and three multimodal large language models: it retains over 99% of original accuracy even after removing 94.4% of visual tokens, achieving a 9.9× reduction in FLOPs and a 6.4× speedup in prefill latency.