LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation

2026-08-12

AI Summary

This work addresses the challenge of sparse and delayed supervision in long-horizon reasoning, which arises when local reflection becomes misaligned with global objectives. To tackle this, the authors propose a memory-controlled reflection modeling framework that incorporates explicit reflection and backtracking actions through an invertible trajectory tree. A novel prospective dual-channel mechanism is introduced to jointly leverage global-view distillation and trajectory-level reinforcement learning (GRPO). Crucially, privileged teacher guidance is strategically focused on reflection and backtracking tokens, effectively aligning local decisions with overall task success. The approach demonstrates significant performance gains over baselines relying solely on outcome-based rewards or self-distillation, particularly in multi-hop retrieval-augmented generation and mathematical reasoning tasks.

Abstract

Large language model agents increasingly rely on long-horizon reasoning to solve complex tasks involving planning, tool use, and memory. A critical capability in such settings is reflection: assessing trajectory progress, identifying missing evidence and unreliable intermediate states, and deciding whether to continue, revise, or abandon the current branch. Learning effective reflection, however, is challenging because reflection is performed locally within the current branch, whereas its utility can only be determined by its contribution to the final trajectory outcome. This local-global mismatch makes outcome-based reinforcement learning provide only local, sparse and delayed supervision for reflective decisions. To solve these, we propose LoongReflect, a training framework that formulates reflection as a memory-control policy. The agent operates over a reversible trajectory tree using explicit reflect and backtrack actions. Reflection consolidates verified facts, missing evidence, and branch-specific risks into working memory, while backtracking removes an unreliable branch from the active context and preserves a concise corrective lesson. To learn this policy, LoongReflect combines two complementary signals through a look-ahead, extragradient-style coordination mechanism. A fast channel distills globally informed reflective behavior from a privileged teacher, with supervision restricted to reflection and backtracking tokens. A slow channel optimizes complete trajectories using outcome-based GRPO, aligning local control decisions with final task success. Experiments on multi-hop retrieval-augmented generation and mathematical reasoning benchmarks demonstrate consistent improvements over outcome-only reinforcement learning and self-distillation baselines.

Problem

• long-horizon reasoning
• reflection
• local-global mismatch
• reinforcement learning
• trajectory outcome

Innovation

• reflection
• global perspective distillation
• trajectory tree
• memory-control policy
• extragradient coordination