FlexiSLM: A Dynamic and Controllable Frame Rate Spoken Language Model

2026-06-30

AI Summary

This work addresses the limitations of existing speech language models that rely on fixed frame rates, which struggle to adapt to the time-varying information density of speech and lack flexibility in trading off generation quality against speed during inference. To overcome these challenges, we propose the first speech language model supporting dynamically controllable frame rates at both input and output stages. Our approach introduces a novel dynamic-frame-rate speech representation and a controllable frame-rate audio tokenizer, implemented within a 7B-scale language model architecture for efficient modeling. Experimental results demonstrate that our method outperforms fixed-frame-rate models such as Qwen2.5-Omni and Kimi-Audio in high-quality generation mode. Moreover, it halves inference time at 6.25 Hz while maintaining good speech quality even at 4.0 Hz, substantially enhancing both efficiency and adaptability.

Abstract

Spoken language models (SLMs) extend LLMs to speech input and output. Existing SLMs represent speech at fixed frame rates (e.g., 25 or 12.5 Hz), ignoring the time-varying information density of speech and offering no flexibility to trade off quality for speed at inference time. Recent audio tokenizer research has proposed dynamic frame rate speech coding, which exploits this non-uniformity and enables two new capabilities: very low average frame rates and frame rate controllability. However, this technique has not yet been applied to SLMs. We introduce Flexible Spoken Language Model (FlexiSLM), the first SLM that supports dynamic and controllable frame rates on both speech input and output. Using dynamic frame rate representations, FlexiSLM outperforms fixed-frame-rate 7B models including Qwen2.5-Omni and Kimi-Audio at its high-quality operating points. We further verify that FlexiSLM can be accurately steered down to 4.0 Hz; at 6.25 Hz, it roughly halves inference time relative to 12.5 Hz while retaining strong speech-to-speech quality. Audio samples are available at https://flexislm.github.io .

Problem

• spoken language model
• fixed frame rate
• dynamic frame rate
• speech representation
• inference efficiency

Innovation

• dynamic frame rate
• controllable speech generation
• spoken language model
• audio tokenization
• inference efficiency