Relative Time Intervals Representation for Word-level Timestamping with Masked Training

2026-08-24

AI Summary

本文通过使用相对时间戳和混合微调策略，增强SpeechLLM对语音内容和时间结构的联合建模能力，提高时间戳预测准确性。

Abstract

Although Speech Large Language Models (SpeechLLMs) excel at speech understanding and generation, their capacity for fine-grained, temporally aligned outputs remains underexplored. Our work addresses this gap by enabling SpeechLLMs to jointly model speech content and temporal structure, effectively transforming them from ``content understanding machines" into ``temporal-aware content understanding machines". Specifically, we replace traditional absolute timestamps with relative timestamps, achieving a more compact vocabulary and stronger generalization capabilities. To efficiently infuse timestamp prediction ability into pre-trained large language models, we introduce a hybrid fine-tuning strategy: full-parameter fine-tuning of the timestamp-augmented embedding layer and language model head, combined with LoRA fine-tuning of the decoder layers. Moreover, we design a masked timestamp training objective, preventing the model from over-relying on ground-truth timestamps, and thereby enhancing robustness against noisy real-world annotations. Extensive experiments demonstrate that our approach achieves significant improvements in timestamp prediction accuracy while maintaining strong speech transcription performance.

Problem

• Speech Large Language Models
• timestamp prediction
• temporal alignment

Innovation

• relative timestamps
• hybrid fine-tuning strategy
• masked timestamp training