THOUGHTSCULPT: Reasoning with Intermediate Revision and Search

2024-04-09

arXiv.org

AI Summary

This work addresses tasks whose outputs decompose into multiple interdependent components. We propose a Monte Carlo Tree Search (MCTS)-based reasoning and search framework that explicitly models “revision actions” as first-class operations in the search space—departing from conventional left-to-right autoregressive generation and enabling dynamic refinement of intermediate outputs. The framework employs a large language model as a learned heuristic evaluator and defines a customizable action space comprising both generation and revision operators. Evaluated on story outline optimization, mini crossword solving, and constrained text generation, our method achieves new state-of-the-art performance: +30% improvement in engagement score, +16% word-level accuracy, and +10% concept coverage. These gains demonstrate substantial advances in both reasoning fidelity and controllability for complex, structured generation tasks.

Technology Category

• Search and Optimization: Learning to Search
• Natural Language Processing: Generation
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We present THOUGHTSCULPT, a general reasoning and search method for tasks with outputs that can be decomposed into components. THOUGHTSCULPT explores a search tree of potential solutions using Monte Carlo Tree Search (MCTS), building solutions one action at a time and evaluating according to any domain-specific heuristic, which in practice is often simply an LLM evaluator. Critically, our action space includes revision actions: THOUGHTSCULPT may choose to revise part of its previous output rather than continuing to build the rest of its output. Empirically, THOUGHTSCULPT outperforms state-of-the-art reasoning methods across three challenging tasks: Story Outline Improvement (up to +30% interestingness), Mini-Crosswords Solving (up to +16% word success rate), and Constrained Generation (up to +10% concept coverage).

Problem

• General reasoning and search method
• Outputs decomposed into components
• Revision actions improve solutions

Innovation

• Monte Carlo Tree Search
• Revision actions integration
• LLM-based heuristic evaluation