WEST: LLM based Speech Toolkit for Speech Understanding, Generation, and Interaction

2025-09-24

AI Summary

To address the high technical barriers, modular fragmentation, and poor extensibility in speech technology, this paper introduces WEST—the first fully LLM-based, unified full-stack speech toolkit. Methodologically, we design an LLM-native speech processing framework that integrates the Hugging Face ecosystem, sequence packing, and multi-stage instruction tuning to enable end-to-end, unified modeling of automatic speech recognition (ASR), text-to-speech (TTS), spoken language understanding (SLU), dialogue, and multimodal interaction. Our contributions are threefold: (1) an open-source, reproducible WEST toolkit—including complete training configurations, pre-trained models, and lightweight deployment solutions; (2) out-of-the-box high-performance speech processing, achieving state-of-the-art or near-state-of-the-art results on ASR, TTS, and SLU benchmarks; and (3) empirical validation of the feasibility and scalability of pure-LLM architectures for end-to-end speech modeling, substantially lowering the adoption barrier for speech technologies.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In this paper, we present WEST(WE Speech Toolkit), a speech toolkit based on a large language model (LLM) for speech understanding, generation, and interaction. There are three key features of WEST: 1) Fully LLM-based: Standing on the shoulders of giants by reusing mature architectures, ecosystems (e.g., Hugging Face), and methods (e.g., sequence packing) from large models. 2) Full-stack: Supports tasks such as recognition, synthesis, understanding, dialogue, and multimodal capabilities, with extensibility to incorporate open-source models. 3) Simple and Stupid: A simple and stupid speech toolkit that everyone can Touch. In addition, WEST provides two types of recipes, models, and experimental results. The first is entirely based on open-source models and open-source data, allowing users to fully reproduce the experiments in this paper and serving as a verification system or minimal system baseline. The second is trained on massive data, offering superior performance so the user can directly apply it out of the box. WEST is publicly avilable at https://github.com/wenet-e2e/west/

Problem

• Developing a speech toolkit using large language models
• Supporting speech understanding, generation, and interaction tasks
• Providing reproducible experiments and ready-to-use models

Innovation

• LLM-based speech toolkit for understanding, generation, interaction
• Full-stack support for recognition, synthesis, dialogue, multimodal tasks
• Simple design with open-source models and massive data training