Test-Time Deep Thinking to Explore Implicit Rules

2026-05-23

AI Summary

This work addresses the challenge of agents repeatedly resorting to trial-and-error in environments governed by implicit rules due to insufficient reasoning capabilities. To overcome this, the authors propose TTExplore, a test-time exploration framework that leverages a large language model within a thinker–actor architecture. The thinker infers implicit rules from interaction history to guide the actor, utilizing task-level scores as sparse reward signals. By employing a trajectory pruning strategy that retains only one critical reasoning node per trajectory and integrating a stable reinforcement learning training mechanism, TTExplore substantially mitigates the issue of reward sparsity. Evaluated across five text-based embodied tasks, TTExplore outperforms baseline methods by an average of 14–19 points, demonstrating both the efficacy and generalization capability of explicitly reasoning about implicit rules.

Abstract

With the continuous advancement of Large Language Models (LLMs), intelligent agents are becoming increasingly vital. However, these agents often fail in environments governed by implicit rules--hidden constraints that cannot be observed directly and must be inferred through interaction. This causes agents to fall into repetitive trial-and-error loops, ultimately leading to task failure. To address this challenge, we propose Test-Time Exploration (TTExplore), a framework where a thinker component analyzes interaction history to infer these implicit rules and guide an actor. Effective exploration in this setting critically depends on the reasoning ability of the thinker. However, evaluating deep reasoning trajectories is inherently unstable and difficult, which poses a major obstacle to effective training. To overcome this issue, we introduce a novel and stable reinforcement learning pipeline. The core idea is to use accurate task-level scores as indirect rewards to bypass the difficulty of evaluating intermediate reasoning, and to retain only a single thinking node per trajectory to alleviate reward sparsity. Using this pipeline, we train a specialized 7B model, Exp-Thinker. Experiments on five text-based embodied tasks show that TTExplore equipped with Exp-Thinker improves baseline agent performance by an average of $14$-$19$ points, demonstrating the effectiveness of explicitly reasoning about implicit rules.

Problem

• implicit rules
• intelligent agents
• test-time exploration
• reasoning
• embodied tasks

Innovation

• Test-Time Exploration
• Implicit Rules
• Reasoning Agent
• Reinforcement Learning
• Large Language Models