Slamming: Training a Speech Language Model on One GPU in a Day

2025-02-19

AI Summary

This work addresses the prohibitively high computational cost hindering academic research on Speech Language Models (SLMs). We propose a lightweight training paradigm: leveraging efficient initialization and a streamlined architecture, coupled with controllable synthetic speech-text data generation and DPO-style preference optimization, enabling high-quality SLM training within 24 hours on a single consumer-grade GPU. Empirical evaluation demonstrates competitive performance against state-of-the-art multi-GPU models across ASR, TTS, and speech understanding tasks, while reducing computational cost to less than 5% of conventional approaches. Our results significantly surpass predictions from traditional scaling laws, redefining the feasibility frontier for SLMs under extreme resource constraints. To foster reproducibility and further research, we fully open-source the code, data, and trained models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We introduce Slam, a recipe for training high-quality Speech Language Models (SLMs) on a single academic GPU in 24 hours. We do so through empirical analysis of model initialisation and architecture, synthetic training data, preference optimisation with synthetic data and tweaking all other components. We empirically demonstrate that this training recipe also scales well with more compute getting results on par with leading SLMs in a fraction of the compute cost. We hope these insights will make SLM training and research more accessible. In the context of SLM scaling laws, our results far outperform predicted compute optimal performance, giving an optimistic view to SLM feasibility. See code, data, models, samples at - https://pages.cs.huji.ac.il/adiyoss-lab/slamming .

Problem

• Efficient SLM training
• Single GPU optimization
• Reduced compute cost

Innovation

• Single GPU training
• Synthetic data optimization
• Cost-effective SLM scaling