CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agents

2026-06-19

AI Summary

This work addresses the performance degradation of task-oriented spoken language understanding systems in noisy environments due to automatic speech recognition (ASR) errors, and the high cost of acquiring speech–semantics aligned annotations for adapting speech language models to new tasks. The authors propose CORTIS, a novel framework that, for the first time, enables end-to-end fine-tuning of speech language models using only textual task supervision signals—eliminating the need for task-specific speech annotations. Experiments based on Qwen2.5-Omni demonstrate that CORTIS matches the performance of conventional ASR–LLM cascade systems on three task-oriented spoken datasets under clean conditions, while significantly outperforming them in noisy settings, particularly exhibiting superior robustness in preserving high-level semantic content.

Abstract

Task-oriented voice agents need to map spoken user requests to structured outputs such as semantic frames, executable actions, and function calls. A common approach is to cascade ASR with a text-based LLM, but transcription errors can propagate to downstream structured output generation, especially under noisy conditions. Spoken language models (SLMs) offer a direct speech-based alternative, yet adapting them to new tasks typically requires paired speech-target annotations. Motivated by this gap, we present CORTIS, a text-only adaptation framework for task-oriented voice agents. CORTIS fine-tunes SLMs using text-form task supervision, enabling speech-based structured output generation at inference time without task-specific speech-target annotations during adaptation. We evaluate CORTIS on two Qwen2.5-Omni backbones and three task-oriented speech datasets, including an in-house product dataset, and compare it with matched ASR-LLM cascades trained with the same text-form task supervision. Results show that CORTIS performs competitively with matched cascades and offers clearer advantages under acoustic degradation, particularly in preserving high-level task semantics. These findings suggest that text-only fine-tuning of SLMs can serve as a practical adaptation strategy for voice agents when paired speech-target data are costly to collect.

Problem

• task-oriented voice agents
• spoken language models
• structured output generation
• speech-target annotations
• acoustic degradation

Innovation

• spoken language models
• text-only adaptation
• task-oriented voice agents
• structured output generation
• acoustic robustness