🤖 AI Summary
Large reasoning models (LRMs) suffer from high time-to-first-token (TTFT) and low inference efficiency due to lengthy implicit chain-of-thought (CoT) reasoning. To address this, we propose MinD (Structured Multi-turn Decomposition), a novel paradigm that explicitly decomposes CoT into controllable, interruptible multi-turn interactions—each generating one reasoning unit and its corresponding answer. Our contributions include: (1) the first explicit, structured, turn-controllable multi-turn reasoning framework; (2) real-time user intervention capability (e.g., pause/resume); and (3) the first application of reinforcement learning (specifically GRPO) to jointly optimize both the number of reasoning turns and answer correctness. MinD integrates supervised fine-tuning (SFT), LLM-based rewriting distillation, multi-turn formatted data construction, and the R1-Distill architecture. Evaluated on MATH, MinD reduces TTFT and output token count by ~70%, while maintaining competitive performance on MATH-500 and AIME24.
📝 Abstract
Large Reasoning Models (LRMs) are criticized for the excessively lengthy Chain-of-Thought (CoT) to derive the final answer, suffering from high first-token and overall latency. Typically, the CoT of LRMs mixes multiple thinking units; each unit attempts to produce a candidate answer to the original query. Hence, a natural idea to improve efficiency is to reduce the unit number. Yet, the fact that the thinking units in vanilla CoT cannot be explicitly managed renders doing so challenging. This paper introduces Multi-Turn Decomposition (MinD) to decode conventional CoT into a sequence of explicit, structured, and turn-wise interactions to bridge the gap. In MinD, the model provides a multi-turn response to the query, where each turn embraces a thinking unit and yields a corresponding answer. The subsequent turns can reflect, verify, revise, or explore alternative approaches to both the thinking and answer parts of earlier ones. This not only makes the answer delivered more swiftly, but also enables explicit controls over the iterative reasoning process (i.e., users may halt or continue at any turn). We follow a supervised fine-tuning (SFT) then reinforcement learning (RL) paradigm to realize MinD. We first rephrase the outputs of an LRM into multi-turn formats by prompting another LLM, and then tune the LRM with such data. Observing that the tuned model tends to consume even more tokens than the original one (probably due to that the multi-turn formats introduce additional answer tokens), we advocate leveraging RL algorithms like GRPO to prioritize correct outputs with fewer turns. Trained on the MATH dataset using R1-Distill models, MinD can achieve up to ~70% reduction in both output token usage and time to first token (TTFT), while maintaining competitive performance on reasoning benchmarks such as MATH-500, AIME24, AMC23, and GPQA-Diamond.