Spoken Language Understanding on Unseen Tasks With In-Context Learning

2025-05-12

AI Summary

Existing zero-/few-shot spoken language understanding (SLU) methods suffer from weak generalization to unseen tasks and heavy reliance on task-specific annotated data. Method: This paper proposes a robust, task-agnostic fine-tuning framework that requires no new task annotations. It introduces randomized label shuffling during pre-fine-tuning to decouple the model from overfitting to fixed label distributions, thereby enhancing adaptability to novel task structures. The approach integrates speech-text multimodal large language models, in-context learning, and a unified zero-/few-shot evaluation protocol. Contribution/Results: Our method achieves significant improvements over state-of-the-art open-source speech-text LLM baselines across diverse unseen SLU tasks—including information extraction, classification, and generation—with an average zero-shot performance gain of 28.6%. To our knowledge, this is the first work to empirically demonstrate that general-purpose speech-text pretrained models alone can enable fully annotation-free, cross-task, and strongly generalizable SLU adaptation.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Spoken language understanding (SLU) tasks involve diverse skills that probe the information extraction, classification and/or generation capabilities of models. In this setting, task-specific training data may not always be available. While traditional task-specific SLU models are unable to cater to such requirements, the speech-text large language models (LLMs) offer a promising alternative with emergent abilities. However, out of-the-box, our evaluations indicate that the zero/few-shot performance of prominent open-source speech-text LLMs on SLU tasks are not up to the mark. In this paper, we introduce a novel approach to robust task-agnostic fine-tuning using randomized class labels. With this proposed fine-tuning, we illustrate that the performance of the speech-text LLMs on an unseen task is significantly improved over standard approaches. Critically, the proposed approach avoids the requirement of task-specific data annotations for enabling new tasks in speech-text LLMs.

Problem

• Improving SLU performance on unseen tasks without task-specific data
• Enhancing zero/few-shot abilities of speech-text LLMs for SLU
• Eliminating need for task-specific annotations in speech-text LLMs

Innovation

• Uses in-context learning for unseen tasks
• Employs randomized class labels for fine-tuning
• Eliminates need for task-specific data annotations