LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model

2025-08-21

AI Summary

Long-standing challenges in speech-language modeling—including architectural fragmentation, non-public training data, and undocumented configurations—have severely hindered reproducibility and systematic evaluation. To address these issues, this work introduces LLaSO, a fully open-source end-to-end framework. Methodologically, it unifies large-scale speech-text alignment, multi-task instruction tuning, end-to-end modeling, and complete training configuration reproduction. Crucially, LLaSO releases, for the first time, 12 million speech-text aligned pairs, 13.5 million instruction-tuning samples, and a standardized benchmark suite. Leveraging only publicly available data, we train LLaSO-Base, a 3.8B-parameter foundation model. Under the unified benchmark, LLaSO-Base achieves a normalized score of 0.72—significantly outperforming prior comparable models. All data, code, and models are openly released, establishing a strong, reproducible baseline and a standardized foundation for collaborative research in speech-language modeling.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The development of Large Speech-Language Models (LSLMs) has been slowed by fragmented architectures and a lack of transparency, hindering the systematic comparison and reproducibility of research. Unlike in the vision-language domain, the LSLM field suffers from the common practice of releasing model weights without their corresponding training data and configurations. To address these critical gaps, we introduce LLaSO, the first fully open, end-to-end framework for large-scale speech-language modeling. LLaSO provides the community with three essential resources: (1) LLaSO-Align, a 12M-instance speech-text alignment corpus; (2) LLaSO-Instruct, a 13.5M-instance multi-task instruction-tuning dataset; and (3) LLaSO-Eval, a reproducible benchmark for standardized evaluation. To validate our framework, we build and release LLaSO-Base, a 3.8B-parameter reference model trained exclusively on our public data. It achieves a normalized score of 0.72, establishing a strong, reproducible baseline that surpasses comparable models. Our analysis reveals that while broader training coverage enhances performance, significant generalization gaps persist on unseen tasks, particularly in pure audio scenarios. By releasing the complete stack of data, benchmarks, and models, LLaSO establishes a foundational open standard to unify research efforts and accelerate community-driven progress in LSLMs. We release the code, dataset, pretrained models, and results in https://github.com/EIT-NLP/LLaSO.

Problem

• Addressing fragmented architectures in large speech-language models
• Solving lack of transparency and reproducibility in LSLM research
• Providing standardized benchmarks for systematic model comparison

Innovation

• Open end-to-end framework for speech-language modeling
• Provides speech-text alignment and instruction-tuning datasets
• Includes reproducible benchmark for standardized evaluation