Stage-adaptive Token Selection for Efficient Omni-modal LLMs

2026-05-19

AI Summary

This work addresses the high computational cost of dense audio-visual tokens in fully multimodal large language models (LLMs), where existing training-free token selection methods fail to adapt to the dynamic shifts in cross-modal importance across different network layers. To overcome this, the authors propose SEATS—a training-free, stage-wise adaptive token selection mechanism. SEATS first removes spatiotemporal redundancy prior to the LLM via attention-weighted diversity-based selection, then progressively prunes tokens within the LLM in stages, dynamically allocating modality-specific retention budgets based on query relevance. Finally, all non-text tokens are discarded after fusion. Experiments on Qwen2.5-Omni and Qwen3-Omni demonstrate that retaining only 10% of audio-visual tokens yields a 9.3× reduction in FLOPs, a 4.8× speedup in prefill latency, and maintains 96.3% of the original model performance.

Abstract

Omni-modal large language models (om-LLMs) achieve unified audio-visual understanding by encoding video and audio into temporally aligned token sequences interleaved at the window level. However, processing these dense non-textual tokens throughout the LLM incurs substantial computational overhead. Although training-free token selection can reduce this cost, existing methods either focus on visual-only inputs or prune om-LLM tokens only before the LLM with fixed per-modality ratios, failing to capture how cross-modal token importance evolves across layers. To address this limitation, we first analyze the layer-wise token dependency of om-LLMs. We find that visual and audio dependencies follow a block-wise pattern and gradually weaken with depth, indicating that many late-layer non-textual tokens become redundant after cross-modal fusion. Motivated by this observation, we propose SEATS, a training-free, stage-adaptive token selection method for efficient om-LLM inference. Before the LLM, SEATS removes spatiotemporal redundancy via attention-weighted diversity selection. Inside the LLM, it progressively prunes tokens across blocks and dynamically allocates the retention budget from temporal windows to modalities using query relevance scores. In late layers, it removes all remaining non-textual tokens once cross-modal fusion is complete. Experiments on Qwen2.5-Omni and Qwen3-Omni demonstrate that SEATS effectively improves inference efficiency. Retaining only 10% of visual and audio tokens, it achieves a 9.3x FLOPs reduction and a 4.8x prefill speedup while preserving 96.3% of the original performance.

Problem

• omni-modal LLMs
• token selection
• computational overhead
• cross-modal fusion
• stage-adaptive

Innovation

• stage-adaptive token selection
• omni-modal LLMs
• cross-modal fusion
• training-free pruning
• query relevance scoring