CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning

2026-07-05

AI Summary

This work addresses the high inference cost and limited acoustic representation of conventional audio captioning systems that rely on fixed audio encoders. To overcome these limitations, the authors propose an efficient encoder-free audio description model that leverages cross-component knowledge distillation: perceptual-layer knowledge from a pretrained audio teacher model is distilled into a lightweight projector, while semantic-layer knowledge is transferred directly into a large language model (LLM). By integrating LoRA adapters and freezing the LLM parameters, the approach enables effective yet computationally efficient knowledge transfer. Evaluated on the AudioCaps and Clotho datasets, the method achieves CIDEr-D scores of 55.4—gaining 12.18 and 5.21 points respectively—approaching the performance ceiling of 66.4 attained by encoder-retaining architectures.

Abstract

Modern automated audio captioning systems pair a frozen audio encoder with a large language model (LLM) via a trainable projector, incurring the encoder's inference cost and bottlenecking the model through its fixed acoustic features. We present CARD, an encoder-free audio captioning model that removes the encoder at inference: a 13.2M projector feeds a frozen LLM with merged LoRA adapters, while the teacher used to train it is discarded. CARD distills a pretrained audio teacher (CLAP-HTSAT) into the model, but rather than injecting it into the LLM alone, it routes the teacher's representations across components: perceptual stages to the projector and semantic stages to the LLM. This placement improves CIDEr-D by +12.18 over an LLM-only distilled model on AudioCaps and by +5.21 on Clotho, reaching 55.4 against a 66.4 encoder-kept upper bound with no encoder at inference, showing that where a teacher's knowledge is placed matters as much as its presence.

Problem

• audio captioning
• encoder-free
• inference cost
• acoustic features bottleneck
• frozen audio encoder

Innovation

• encoder-free
• cross-component distillation
• audio captioning
• knowledge distillation
• frozen LLM