Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Models

2026-07-13

AI Summary

This work addresses the limited capability of existing large audio language models in perceiving fine-grained non-semantic acoustic attributes, such as vocal emotion. The authors propose a training-free, label-free inference-time intervention method that identifies and amplifies a small subset of neurons most sensitive to acoustic information by comparing their activation patterns in response to real speech versus noise reference signals. This approach enables, for the first time, neuron-level precise intervention within the audio encoder itself, revealing the critical roles of encoder depth and neuronal selectivity in acoustic perception. Experimental results demonstrate substantial improvements, with average accuracy gains of 25.7, 21.4, and 9.7 percentage points on Audio-Flamingo-3, Qwen2.5-Omni, and Kimi-Audio, respectively, significantly outperforming intervention strategies applied at the decoder or within the language model.

Abstract

Large audio-language models (LALMs) often underperform on fine-grained, non-semantic attributes of speech, such as a speaker's emotion, despite strong performance on speech content. Improving this without the cost of retraining calls for an effective inference-time intervention, yet most existing methods intervene only after the audio encoder and operate at a relatively coarse granularity. The encoder itself, where acoustic information is first extracted from the waveform, remains largely unexplored, especially at the level of individual neurons. We introduce IAAN, Identifying and Amplifying Acoustic Neurons, a training-free and label-free method that scores each feed-forward neuron in the audio encoder by contrasting its activation on the real waveform with that on a noise reference lacking the real audio's acoustic information. IAAN then amplifies a small set of the highest-scoring neurons at inference. Across ten non-semantic speech attributes, IAAN improves average accuracy by 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5-Omni, and 9.7 on Kimi-Audio. It also improves a model already explicitly fine-tuned to prioritize acoustic evidence. In controlled comparisons, both the encoder locus and neuron-level selectivity prove necessary for this gain. Intervening after the encoder, at the decoding side or inside the language model, yields little to no improvement, or even deteriorates accuracy. The improvement also depends on which specific neurons are amplified, not merely on their number, confirming that IAAN's acoustic score succeeds in identifying the neurons that matter. These results show that a small, precisely targeted intervention inside the audio encoder is an effective and largely untapped way to strengthen the acoustic understanding of LALMs, opening a new direction for inference-time methods that improve acoustic perception through neuron-level access to the encoder.

Problem

• acoustic perception
• non-semantic speech attributes
• audio-language models
• neuron-level intervention
• encoder-side modulation

Innovation

• neuron-level intervention
• acoustic perception
• training-free method
• audio encoder
• large audio-language models