Evidence-State Rewards for Long-Context Reasoning

2026-07-02

AI Summary

This work addresses the challenge in long-context reasoning where models struggle to dynamically locate, revise, and integrate dispersed evidence. Existing reinforcement learning approaches offer only coarse-grained rewards based on final answers or static evidence selections. To overcome this limitation, the authors propose Maven, a novel framework that models evidence as an editable memory state and introduces an answer-conditioned evidence-state value function within reinforcement learning. Maven further designs action-level fine-grained rewards: add actions are guided by marginal gain and hindsight contribution, link actions leverage evidence synergy, and delete actions measure the improvement in answer support after removing distractors. Trained via GRPO on large language models such as Llama and Qwen, Maven significantly outperforms baselines on LongBench v2, LongReason, and RULER benchmarks, generating more comprehensive evidence sets while substantially reducing distractor retention.

Abstract

Long-context reasoning requires models to locate, revise, and synthesize evidence distributed across lengthy inputs. Existing long-context RL methods usually reward final answers or static evidence extraction, offering little feedback on how intermediate actions change the model's evidence state. We propose Maven, a reinforcement learning framework with an editable evidence memory. Maven defines an answer-conditioned evidence-state value and rewards action-level state transitions: add actions are credited by marginal gain and hindsight contribution, link actions by evidence synergy, and drop actions by improved answer support after removing misleading evidence. These rewards are assigned to the corresponding action spans in GRPO. Across Llama and Qwen models on LongBench v2, LongReason, and RULER, Maven outperforms outcome-only RL and evidence-identification baselines, producing more sufficient evidence sets and lower distractor retention. Our results show that long-context RL benefits from optimizing stateful evidence navigation rather than one-shot evidence extraction.

Problem

• long-context reasoning
• evidence state
• reinforcement learning
• intermediate actions
• evidence navigation

Innovation

• evidence-state rewards
• editable evidence memory
• long-context reasoning
• action-level reinforcement learning
• stateful evidence navigation