Efficient Thought Space Exploration through Strategic Intervention

2025-11-13

AI Summary

Exhaustive sampling in large language model (LLM) inference incurs high computational cost and low efficiency. Method: This paper proposes Hint-Practice Reasoning (HPR), a novel framework that dynamically identifies critical decision points using a newly introduced Distribution Inconsistency Reduction (DIR) metric, enabling strategic intervention within a tree-structured probability space. HPR employs a lightweight surrogate model for primary reasoning while leveraging a stronger LLM solely for probabilistic guidance—achieving collaborative inference. The method integrates decoding trajectory analysis, prompt-driven guidance, dynamic path pruning and reweighting, and iterative optimization. Results: On arithmetic and commonsense reasoning benchmarks, HPR achieves up to 5.1% higher accuracy than self-consistency and MCTS baselines while consuming only 20% of their decoding tokens, maintaining comparable or lower computational overhead. This yields significant improvements in the trade-off between inference efficiency and accuracy.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Probabilistic Programming
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

While large language models (LLMs) demonstrate emerging reasoning capabilities, current inference-time expansion methods incur prohibitive computational costs by exhaustive sampling. Through analyzing decoding trajectories, we observe that most next-token predictions align well with the golden output, except for a few critical tokens that lead to deviations. Inspired by this phenomenon, we propose a novel Hint-Practice Reasoning (HPR) framework that operationalizes this insight through two synergistic components: 1) a hinter (powerful LLM) that provides probabilistic guidance at critical decision points, and 2) a practitioner (efficient smaller model) that executes major reasoning steps. The framework's core innovation lies in Distributional Inconsistency Reduction (DIR), a theoretically-grounded metric that dynamically identifies intervention points by quantifying the divergence between practitioner's reasoning trajectory and hinter's expected distribution in a tree-structured probabilistic space. Through iterative tree updates guided by DIR, HPR reweights promising reasoning paths while deprioritizing low-probability branches. Experiments across arithmetic and commonsense reasoning benchmarks demonstrate HPR's state-of-the-art efficiency-accuracy tradeoffs: it achieves comparable performance to self-consistency and MCTS baselines while decoding only 1/5 tokens, and outperforms existing methods by at most 5.1% absolute accuracy while maintaining similar or lower FLOPs.

Problem

• Reducing computational costs of exhaustive sampling in LLM reasoning
• Identifying critical decision points causing reasoning deviations
• Optimizing efficiency-accuracy tradeoffs in language model inference

Innovation

• Hint-Practice Reasoning framework with two components
• Distributional Inconsistency Reduction identifies critical intervention points
• Iterative tree updates reweight promising reasoning paths