Problem
Research questions and friction points this paper is trying to address.
large language models
vocabulary head
discrete tokens
A Model with No Head and Many Thoughts
🤖 AI Summary
为解决语言模型推理时计算成本高和离散化问题，提出使用轻量级投影器替代传统头部的方法，在连续空间中进行推理，实验显示该方法有效提高了性能。
📝 Abstract
Large language models decode by projecting hidden states through a large vocabulary head at every step. This operation is computationally costly and forces all reasoning to be expressed in discrete tokens. We introduce Soft Latent Thinking, a method that replaces the LM head during reasoning with a lightweight projector, enabling autoregressive rollout in embedding space where reasoning steps remain continuous rather than tokenized. Experiments on DeepSeek-Qwen-1.5B and LLaMA-3.2-3B show that Soft Latent Thinking consistently improves pass@k across all k while reducing per-step compute during chain-of-thought. Our method achieves the highest pass@32 among all soft-thinking approaches, demonstrating that effective reasoning can be carried out in continuous space without discrete token generation.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Soft Latent Thinking
continuous reasoning
lightweight projector
embedding space
🔎 Similar Papers
No similar papers found.
