Second Thought: Reasoning in Parallel as LLM Agents Act and Observe

2026-08-13

AI Summary

This study addresses the computational inefficiency in ReAct agents caused by idle reasoning during action execution and observation phases. We propose Second Thought, a training-free framework that pioneers parallel auxiliary reasoning generation during environment feedback latency windows, merging results upon observation return to offload computation from the primary decoding path. Experiments demonstrate that this approach accelerates agent decision-making without additional training, reducing main-thread decoding by an average of 20% (up to 43%) while maintaining or improving Pass@1 scores by up to 12.4 points. Second Thought significantly outperforms baselines with equivalent computational budgets, effectively achieving simultaneous optimization of both inference efficiency and task performance.

Abstract

LLM agents in the ReAct paradigm alternate between reasoning, acting, and observing, but deliberate reasoning is confined to the Thought phase: while the agent serializes an action and waits for the environment, its reasoning is frozen. We identify this recurring interval for Action and Observation as a reasoning idle window and ask whether it can host additional reasoning in parallel that serves future turns. Therefore, we propose Second Thought, a training-free inference framework that forks four auxiliary branches the instant each Thought phase concludes, decodes them concurrently with the main loop, and merges the generated thoughts back when the environment observation arrives. In this way, Second Thought relocates the added reasoning off the main thread's sequential decoding path. Across three agentic benchmarks and three reasoning LLMs, Second Thought lowers the average turn count in all nine (model,benchmark) pairs and reduces main thread decoding in six of them by up to 43% (roughly 20% on average among those settings), while leaving it essentially unchanged in a seventh; Pass@1 shows no significant change in seven of nine pairs and the two significant differences are +12.4 and +10.2 points. Against a compute-matched control that forces an equivalent budget onto the main thread's own reasoning, it attains strictly higher Pass@1 with 1.3 to 3.2 less sequential decoding in all four settings where the control applies.

Problem

• LLM Agents
• ReAct Paradigm
• Reasoning Idle Window
• Parallel Reasoning

Innovation

• Parallel Reasoning
• Training-free Inference
• Reasoning Idle Window
• LLM Agents
• Concurrent Decoding