MoWE-Audio: Multitask AudioLLMs with Mixture of Weak Encoders

2024-09-10

arXiv.org

AI Summary

Existing AudioLLMs rely on a single pre-trained audio encoder, whose fixed representation capacity limits generalization across diverse audio tasks. To address this, we propose the Mixture of Weak Encoders (MoWE), which replaces the monolithic encoder with multiple lightweight audio encoders and employs a learnable gating mechanism to dynamically activate a task-adaptive subset of them—while keeping the large language model frozen. This enables low-overhead, task-specific feature enhancement without architectural or parametric overhead on the LLM. MoWE introduces, for the first time, a “mixture of weak encoders” paradigm that breaks the representational bottleneck by jointly optimizing expressiveness and efficiency: it incurs less than 3% additional parameters while significantly increasing representational diversity. Evaluated on a cross-domain multi-task audio understanding benchmark, MoWE achieves an average accuracy gain of +4.2% over strong single-encoder baselines, demonstrating superior adaptability and robustness.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Search and Optimization: Learning to Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The rapid advancements in large language models (LLMs) have significantly enhanced natural language processing capabilities, facilitating the development of AudioLLMs that process and understand speech and audio inputs alongside text. Existing AudioLLMs typically combine a pre-trained audio encoder with a pre-trained LLM, which are subsequently finetuned on specific audio tasks. However, the pre-trained audio encoder has constrained capacity to capture features for new tasks and datasets. To address this, we propose to incorporate mixtures of `weak' encoders (MoWE) into the AudioLLM framework. MoWE supplements a base encoder with a pool of relatively light weight encoders, selectively activated based on the audio input to enhance feature extraction without significantly increasing model size. Our empirical results demonstrate that MoWE effectively improves multi-task performance, broadening the applicability of AudioLLMs to more diverse audio tasks.

Problem

• Enhance feature extraction in AudioLLMs.
• Improve multi-task performance with MoWE.
• Broaden applicability to diverse audio tasks.

Innovation

• Mixture of Weak Encoders
• Enhance feature extraction
• Improve multi-task performance