Muyan-TTS: A Trainable Text-to-Speech Model Optimized for Podcast Scenarios with a $50K Budget

2025-04-27

AI Summary

Current open-source TTS models lack podcast-specific architectural design, complete training codebases, and efficient inference support, hindering real-world deployment. This work introduces the first open-source, trainable, podcast-optimized TTS model, integrating an LLM-based speech synthesis architecture with pretraining on over 100,000 hours of podcast audio. It enables zero-shot voice synthesis and speaker adaptation within minutes. We propose a novel end-to-end training pipeline, a lightweight inference engine, and a low-latency deployment framework—achieving state-of-the-art performance in naturalness, speaker similarity, and inference speed. All code, pretrained model weights, and data processing scripts are publicly released. The system supports real-time TTS inference on a single GPU and incurs controllable training costs (~$50K), significantly enhancing accessibility and engineering practicality of TTS in voice-interaction applications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Speech
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent advancements in text-to-speech (TTS) models have been driven by the integration of large language models (LLMs), enhancing semantic comprehension and improving speech naturalness. However, existing LLM-based TTS models often lack open-source training code and efficient inference acceleration frameworks, limiting their accessibility and adaptability. Additionally, there is no publicly available TTS model specifically optimized for podcast scenarios, which are in high demand for voice interaction applications. To address these limitations, we introduce Muyan-TTS, an open-source trainable TTS model designed for podcast applications within a $50,000 budget. Our model is pre-trained on over 100,000 hours of podcast audio data, enabling zero-shot TTS synthesis with high-quality voice generation. Furthermore, Muyan-TTS supports speaker adaptation with dozens of minutes of target speech, making it highly customizable for individual voices. In addition to open-sourcing the model, we provide a comprehensive data collection and processing pipeline, a full training procedure, and an optimized inference framework that accelerates LLM-based TTS synthesis. Our code and models are available at https://github.com/MYZY-AI/Muyan-TTS.

Problem

• Lack of open-source LLM-based TTS training code
• No podcast-optimized TTS model publicly available
• High cost and limited accessibility of current TTS models

Innovation

• Open-source trainable TTS model for podcasts
• Pre-trained on 100,000 hours of podcast audio
• Optimized inference framework for LLM-based TTS