Escaping the Procrustean Bed: Groupwise Orthogonal Connectors for Audio-Language Models

2026-07-07

AI Summary

This work addresses the vector collapse problem in audio language models, where the use of a Q-Former connector to compress speech encoder outputs often causes representations to align along a single direction, thereby discarding paralinguistic information such as speaker identity, gender, and prosody. To mitigate this issue, the authors propose ORCA, the first method to introduce an inter-group orthogonality constraint: queries are partitioned into multiple groups, and their outputs are enforced to point in distinct directions, effectively restoring paralinguistic diversity. Built upon a Querying Transformer, ORCA employs a grouped orthogonal connector that substantially alleviates output collapse. Experiments demonstrate that ORCA achieves 75.2% accuracy on the SAKURA multi-hop reasoning task—surpassing the 4B baseline by 26.4 percentage points—while reducing query redundancy in the connector by 12× and increasing cross-speaker representation variance by 75×.

Abstract

Audio-language models compress a speech encoder's output through a Querying Transformer (Q-Former) connector before feeding it to a large language model. We identify two failures in this compression. The connector's output vectors collapse to a single direction, and different speakers produce nearly indistinguishable outputs, with paralinguistic cues such as speaker identity, gender, and prosody lost along the way. Our method, ORCA, reverses this collapse by splitting the queries into groups whose outputs are constrained to point in different directions. On SAKURA multi-hop reasoning, ORCA gains 26.4 points over an identically trained 4B baseline, reaching 75.2% (vs. 49.0% for the 8B Audio Flamingo-3). At the connector level, the same change cuts query redundancy by 12x and raises cross-speaker variance by 75x.

Problem

• audio-language models
• vector collapse
• speaker identity
• paralinguistic cues
• query redundancy

Innovation

• groupwise orthogonal connectors
• audio-language models
• query collapse
• paralinguistic preservation
• Q-Former