AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression

2026-06-23

AI Summary

Long-form audio-visual understanding is hindered by the limited context length of large language models and redundancy across modalities. To address this, this work proposes AVOC, a framework that introduces a learnable retrieval-based token compression module between the multimodal encoder and the large language model. The approach formulates compression as selecting the most query-relevant subset of compact tokens under a fixed context budget. It uniquely integrates relevance, importance, and diversity criteria from information retrieval into audio-visual token compression, all within an end-to-end trainable pipeline. Experiments demonstrate that AVOC outperforms the strongest baseline by 4.9 and 5.5 points on OmniVideoBench and LVOmniBench, respectively, and maintains robust performance on hour-long “needle-in-a-haystack” audio-visual tasks.

Abstract

Multimodal Large Language Models have achieved remarkable progress in short-form audio-video understanding, yet long-form audio-video comprehension remains challenged by limited context windows and severe information redundancy. To address these bottlenecks, we propose AVOC, a framework for long-form audio-video understanding in Omni-modal Large Language Models. AVOC introduces a learnable token compression module between the modality encoders and the LLM backbone. We reframe multimodal token compression as a top-$K$ retrieval problem: given a fixed context budget, the module must retrieve a compact subset of tokens that best supports answering the user query. We draw inspiration from three classical Information Retrieval criteria for selecting informative units from a large candidate pool: relevance, importance, and diversity. AVOC instantiates each criterion as a tailored mechanism for audio-video understanding, and integrates them into a unified retrieval-style compression pipeline. Experiments show that AVOC achieves state-of-the-art performance on long-form audio-video benchmarks, surpassing the second-best model by 4.9 and 5.5 points in average accuracy on OmniVideoBench and LVOmniBench, respectively. Moreover, AVOC maintains robust performance on Audio-Video Needle-in-a-Haystack task at durations up to one hour.

Problem

• long-form audio-video understanding
• context window limitation
• information redundancy
• multimodal token compression

Innovation

• token compression
• audio-video understanding
• multimodal retrieval
• Omni-modal LLMs
• long-form video