Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models

2026-08-10

AI Summary

This work addresses the high computational cost of processing long audio-visual sequences in multimodal large language models and the degradation of cross-modal temporal consistency caused by existing token-wise compression methods. The authors propose Omni2LoRA, a two-stage framework that first compresses multimodal context into low-rank adapter parameters via coherence-preserving contextual distillation, entirely bypassing the token bottleneck, and then employs a discrete rank allocation strategy to optimize under a fixed parameter budget. This approach is the first to encode multimodal context into reusable parameter states, maintaining cross-modal consistency at 75% compression and eliminating the need to reload original tokens during inference. Evaluated on four audio-visual question-answering benchmarks, Omni2LoRA achieves an 8–12% average accuracy gain over the strongest baseline under a 30% rank budget, reduces first-token latency by up to 12×, and attains amortized query latency below 0.5 seconds after multiple queries.

Abstract

Omnimodal language models (OLMs) enable unified audio-visual understanding, but processing long joint token sequences makes inference computationally prohibitive. While recent token compression methods attempt to alleviate this burden, compressing modalities in isolation often destroys the temporal cross-modal anchors necessary for coherent reasoning. We introduce Omni2LoRA, a two-stage framework for efficient parametric memory compression via coherence-preserving context distillation that bypasses the token bottleneck entirely. First, a Perceiver hypernetwork processes intermediate representations from a frozen OLM to encode the multimodal context into a full-rank Low-Rank Adaptation (LoRA) adapter in a single forward pass. To prevent the resulting parameter footprint from scaling linearly with recording length, we optimize a discrete rank allocation policy via Group Relative Policy Optimization (GRPO) that uses a modality-ablated counterfactual reward to explicitly penalize the loss of audio-visual coherence, forcing the model to allocate its fixed sub-linear rank budget to synergistic cross-modal anchors rather than isolated visual features. Across three omnimodal backbones, Omni2LoRA operating at a 30% rank budget outperforms direct full-context inference and strong token-compression baselines (OmniZip, OMAC, O-MARC) on four audio-visual question answering benchmarks, improving average accuracy by 8-12% over the strongest baseline and remaining stable under compression ratios as tight as 75%, where token-pruning methods degrade sharply. By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero, cutting per-query Time to First Token (TTFT) by up to 12x relative to full-context inference and amortizing to under 0.5s after a handful of queries.

Problem

• omnimodal language models
• token compression
• cross-modal coherence
• computational efficiency
• multimodal reasoning

Innovation

• Omni2LoRA
• coherence-preserving compression
• parametric memory
• Low-Rank Adaptation (LoRA)
• multimodal reasoning