Cycle-Consistent Search: Question Reconstructability as a Proxy Reward for Search Agent Training

2026-04-14

AI Summary

Current training of search agents heavily relies on gold supervision signals—such as ground-truth answers—that are difficult to scale. This work proposes a novel training framework that eliminates the need for such gold supervision by introducing cycle consistency into search agent optimization for the first time. The approach leverages question reconstruction as a proxy reward and incorporates an information bottleneck—achieved through strategies like removing final answers and masking named entities—to ensure that reconstruction depends on the informational completeness of the retrieval trajectory rather than superficial linguistic cues. Evaluated across multiple question-answering benchmarks, the method achieves performance comparable to supervised approaches and significantly outperforms existing unsupervised methods.

Technology Category

• Search and Optimization: Learning to Search
• Natural Language Processing: Question Answering
• Constraint Satisfaction and Optimization: Search

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reinforcement Learning (RL) has shown strong potential for optimizing search agents in complex information retrieval tasks. However, existing approaches predominantly rely on gold supervision, such as ground-truth answers, which is difficult to scale. To address this limitation, we propose Cycle-Consistent Search (CCS), a gold-supervision-free framework for training search agents, inspired by cycle-consistency techniques from unsupervised machine translation and image-to-image translation. Our key hypothesis is that an optimal search trajectory, unlike insufficient or irrelevant ones, serves as a lossless encoding of the question's intent. Consequently, a high-quality trajectory should preserve the information required to accurately reconstruct the original question, thereby inducing a reward signal for policy optimization. However, naive cycle-consistency objectives are vulnerable to information leakage, as reconstruction may rely on superficial lexical cues rather than the underlying search process. To reduce this effect, we apply information bottlenecks, including exclusion of the final response and named entity recognition (NER) masking of search queries. These constraints force reconstruction to rely on retrieved observations together with the structural scaffold, ensuring that the resulting reward signal reflects informational adequacy rather than linguistic redundancy. Experiments on question-answering benchmarks show that CCS achieves performance comparable to supervised baselines while outperforming prior methods that do not rely on gold supervision. These results suggest that CCS provides a scalable training paradigm for training search agents in settings where gold supervision is unavailable.

Problem

• search agent
• reinforcement learning
• gold supervision
• information retrieval
• unsupervised training

Innovation

• Cycle-Consistent Search
• gold-supervision-free
• question reconstructability
• information bottleneck
• reinforcement learning