DynaAct: Large Language Model Reasoning with Dynamic Action Spaces

2025-11-11

AI Summary

To address scalability and efficiency bottlenecks in constructing candidate action spaces for complex reasoning tasks, this paper proposes a dynamic, compact action space construction method. The core innovation lies in the first application of submodular function optimization to sequential decision-making settings, jointly modeling action utility and diversity. Leveraging problem sketches generated by large language models, the method employs a greedy algorithm to select high-quality candidate actions in real time—eliminating reliance on manual action specification or exhaustive search and significantly reducing inference latency. Evaluated on six standard benchmarks, the approach consistently improves reasoning performance—including accuracy and success rate—while maintaining low computational overhead. The implementation is publicly available.

Technology Category

• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Sequential Decision Making
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In modern sequential decision-making systems, the construction of an optimal candidate action space is critical to efficient inference. However, existing approaches either rely on manually defined action spaces that lack scalability or utilize unstructured spaces that render exhaustive search computationally prohibitive. In this paper, we propose a novel framework named extsc{DynaAct} for automatically constructing a compact action space to enhance sequential reasoning in complex problem-solving scenarios. Our method first estimates a proxy for the complete action space by extracting general sketches observed in a corpus covering diverse complex reasoning problems using large language models. We then formulate a submodular function that jointly evaluates candidate actions based on their utility to the current state and their diversity, and employ a greedy algorithm to select an optimal candidate set. Extensive experiments on six diverse standard benchmarks demonstrate that our approach significantly improves overall performance, while maintaining efficient inference without introducing substantial latency. The implementation is available at https://github.com/zhaoxlpku/DynaAct.

Problem

• Automatically constructs compact action spaces for sequential reasoning
• Overcomes limitations of manual or unstructured action space designs
• Enhances complex problem-solving while maintaining efficient inference

Innovation

• Automatically constructs compact action spaces using LLMs
• Formulates submodular function balancing utility and diversity
• Employs greedy algorithm for optimal candidate selection