The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models

2025-03-04

AI Summary

Large language models (LLMs) often rely heavily on labeled data or computationally expensive sampling strategies—such as rejection sampling—to improve reasoning capabilities, posing significant resource bottlenecks. Method: This paper proposes Unsupervised Prefix Fine-Tuning (UPFT), the first method to exploit the “Prefix Self-Consistency” phenomenon: it fine-tunes only an extremely short shared initial prefix (as few as 8 tokens) from reasoning paths, requiring neither labeled data nor posterior sampling. UPFT integrates prefix truncation, self-consistency modeling, and parameter-efficient optimization while preserving the model’s original knowledge structure. Results: Experiments demonstrate that UPFT matches the performance of supervised rejection sampling fine-tuning (RSFT) across multiple mainstream reasoning benchmarks, while reducing training time by 75% and inference sampling overhead by 99%, thereby overcoming critical resource constraints of conventional approaches.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Improving the reasoning capabilities of large language models (LLMs) typically requires supervised fine-tuning with labeled data or computationally expensive sampling. We introduce Unsupervised Prefix Fine-Tuning (UPFT), which leverages the observation of Prefix Self-Consistency -- the shared initial reasoning steps across diverse solution trajectories -- to enhance LLM reasoning efficiency. By training exclusively on the initial prefix substrings (as few as 8 tokens), UPFT removes the need for labeled data or exhaustive sampling. Experiments on reasoning benchmarks show that UPFT matches the performance of supervised methods such as Rejection Sampling Fine-Tuning, while reducing training time by 75% and sampling cost by 99%. Further analysis reveals that errors tend to appear in later stages of the reasoning process and that prefix-based training preserves the model's structural knowledge. This work demonstrates how minimal unsupervised fine-tuning can unlock substantial reasoning gains in LLMs, offering a scalable and resource-efficient alternative to conventional approaches.

Problem

• Enhance reasoning efficiency in large language models
• Eliminate need for labeled data or exhaustive sampling
• Reduce training time and sampling cost significantly

Innovation

• Unsupervised Prefix Fine-Tuning (UPFT) method
• Trains on initial prefix substrings only
• Reduces training time and sampling cost significantly