Constraining to Generalize: Subspace Tuning for Few-shot Generalization of Audio-Language Models

2026-06-16

AI Summary

This work addresses the base-new class trade-off in few-shot fine-tuning of audio-language models, where zero-shot drift in the text embedding space often improves performance on seen classes at the expense of generalization to unseen ones. To mitigate this issue, the authors propose Subspace Tuning (SubT), a novel framework that introduces geometric constraints into few-shot adaptation for the first time. SubT employs structured subspace parameterization to limit embedding deformation and incorporates a residual anchoring mechanism to stabilize adaptation relative to the zero-shot prior. During inference, a subspace-aware gating strategy suppresses negative transfer for weakly aligned unseen classes. Without requiring backpropagation through the text encoder, SubT achieves significant improvements in few-shot generalization across eleven audio benchmarks while maintaining computational efficiency.

Abstract

Few-shot adaptation of pretrained Audio--Language Models (ALMs) often improves seen-class performance at the cost of unseen-class generalization, leading to the base-to-new trade-off. We attribute this failure to zero-shot drift in the text embedding space: few-shot tuning can distort inter-class structure and move adapted embeddings far from their pretrained anchors. We therefore propose Subspace Tuning (SubT), a geometry-constrained adaptation framework with two complementary controls on drift. Structured Subspace Parameterization limits structural deformation, and Residual Anchoring stabilizes adaptation around the zero-shot prior. At inference time, Subspace-aware Gating further suppresses negative transfer for weakly aligned unseen classes. Across 11 audio benchmarks, SubT delivers strong few-shot generalization while remaining efficient, operating directly on precomputed text embeddings without text-encoder backpropagation.

Problem

• few-shot generalization
• audio-language models
• zero-shot drift
• base-to-new trade-off
• text embedding space

Innovation

• Subspace Tuning
• few-shot generalization
• audio-language models
• zero-shot drift
• geometry-constrained adaptation