TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantization

2025-08-31

AI Summary

To address the high parameter count and computational overhead of Transformer-based music generation models—hindering their deployment on resource-constrained edge devices—this paper proposes an efficient on-device music generation framework. First, we introduce a phase-mixed distillation strategy combining bidirectional and skewed KL divergence to enhance the student model’s fidelity in capturing the teacher’s output distribution. Second, we propose an adaptive mixed-precision quantization scheme that preserves higher precision in critical layers to maintain audio quality. To our knowledge, this is the first work enabling cloud-free, high-fidelity music generation directly on mobile devices. Evaluated on MusicGen-Small, our method reduces model size by 55% while retaining 93% of the original performance; measured inference latency is under 300 ms. The optimized model is successfully deployed on mainstream smartphones, enabling real-time, high-quality, fully local music generation.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: Generation
• Computer Vision: Diffusion Models for Vision

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The success of the generative model has gained unprecedented attention in the music generation area. Transformer-based architectures have set new benchmarks for model performance. However, their practical adoption is hindered by some critical challenges: the demand for massive computational resources and inference time, due to their large number of parameters. These obstacles make them infeasible to deploy on edge devices, such as smartphones and wearables, with limited computational resources. In this work, we present TinyMusician, a lightweight music generation model distilled from MusicGen (a State-of-the-art music generation model). TinyMusician integrates two innovations: (i) Stage-mixed Bidirectional and Skewed KL-Divergence and (ii) Adaptive Mixed-Precision Quantization. The experimental results demonstrate that TinyMusician retains 93% of the MusicGen-Small performance with 55% less model size. TinyMusician is the first mobile-deployable music generation model that eliminates cloud dependency while maintaining high audio fidelity and efficient resource usage

Problem

• Reducing computational demands for on-device music generation
• Overcoming large parameter obstacles in edge device deployment
• Maintaining audio fidelity while eliminating cloud dependency

Innovation

• Knowledge distillation for model compression
• Mixed precision quantization for efficiency
• On-device deployment without cloud dependency