Fast Quiet-STaR: Thinking Without Thought Tokens

2025-05-23

AI Summary

Large language models (LLMs) incur substantial inference overhead in complex reasoning due to reliance on explicit chain-of-thought (CoT) tokens. To address this, we propose an **implicit reasoning compression method driven by curriculum learning**, which progressively reduces the number of reasoning tokens during training, thereby internalizing token-level reasoning capabilities into the standard next-token prediction architecture—eliminating the need for explicit CoT generation at inference time. Our approach integrates curriculum learning, reinforcement learning–based fine-tuning, and implicit reasoning modeling. Evaluated on Mistral-7B and Qwen2.5-7B, it improves average accuracy by 9.0% and 5.7%, respectively, with zero increase in inference latency, outperforming Quiet STaR significantly. This work achieves, for the first time, seamless unification of reasoning and standard language modeling, establishing a new paradigm for efficient, scalable complex reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Language Models (LLMs) have achieved impressive performance across a range of natural language processing tasks. However, recent advances demonstrate that further gains particularly in complex reasoning tasks require more than merely scaling up model sizes or training data. One promising direction is to enable models to think during the reasoning process. Recently, Quiet STaR significantly improves reasoning by generating token-level thought traces, but incurs substantial inference overhead. In this work, we propose Fast Quiet STaR, a more efficient reasoning framework that preserves the benefits of token-level reasoning while reducing computational cost. Our method introduces a curriculum learning based training strategy that gradually reduces the number of thought tokens, enabling the model to internalize more abstract and concise reasoning processes. We further extend this approach to the standard Next Token Prediction (NTP) setting through reinforcement learning-based fine-tuning, resulting in Fast Quiet-STaR NTP, which eliminates the need for explicit thought token generation during inference. Experiments on four benchmark datasets with Mistral 7B and Qwen2.5 7B demonstrate that Fast Quiet-STaR consistently outperforms Quiet-STaR in terms of average accuracy under the same inference time budget. Notably, Fast Quiet-STaR NTP achieves an average accuracy improvement of 9% on Mistral 7B and 5.7% on Qwen2.5 7B, while maintaining the same inference latency. Our code will be available at https://github.com/huangwei200012/Fast-Quiet-STaR.

Problem

• Improving LLM reasoning efficiency without thought tokens
• Reducing computational cost in token-level reasoning
• Enhancing accuracy while maintaining inference speed

Innovation

• Curriculum learning reduces thought tokens gradually
• Reinforcement learning fine-tunes for next token prediction
• Efficient reasoning with lower computational cost