Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference

2026-06-30

AI Summary

This work addresses the high computational cost of visual token processing in multimodal large language model (MLLM) inference, where existing acceleration methods often sacrifice critical information due to coarse-grained pruning. Observing that visual computations in later layers have minimal impact on final outputs, the authors decouple Transformer layers into attention and feed-forward network (FFN) operators and propose a dynamic, operator-level, layer-dependent skipping mechanism. This approach enables fine-grained omission of redundant computations while preserving the full visual token sequence. Evaluated across three MLLM architectures and ten visual question answering (VQA) benchmarks, the method reduces inference FLOPs by 33.7% on Qwen3-VL with only a 0.5% performance drop.

Abstract

Multimodal large language models (MLLMs) increasingly process long visual-token sequences, increasing the overall inference computation. Existing acceleration methods usually remove visual tokens or skip visual-token updates in entire layers, but these coarse strategies may discard fine-grained evidence or suppress useful operators together with redundant ones. In this paper, we study visual-token computation from an answer-observable perspective and find that late visual-token updates can remain large while having little effect on answer-token representations. Motivated by this answer-silent redundancy, we decompose each Transformer layer into attention and FFN operators and show that useful visual computation is often operator-dominant and layer-dependent. We propose an operator-level visual-token skipping framework that preserves the full visual-token sequence while selectively bypassing redundant attention, FFN, or both. Experiments across three MLLM architectures and 10 VQA benchmarks show that our method achieves strong efficiency-accuracy trade-offs, reducing \textbf{33.7\%} TFLOPs on Qwen3-VL while retaining \textbf{99.5\%} of the vanilla model performance.

Problem

• multimodal LLM
• visual-token skipping
• inference efficiency
• operator-level redundancy
• VQA

Innovation

• operator-level skipping
• visual-token redundancy
• multimodal LLM acceleration
• answer-silent redundancy
• efficient inference