Trace as State: Reasoning Traces as Conditional States for Long-Context Transformers

2026-09-02

AI Summary

为了解决长上下文推理中的信息处理问题，本文提出将推理痕迹作为状态前置的方法，通过实验验证该方法优于将痕迹后置。

Abstract

Transformers process information causally, but long-context reasoning may depend on task state discovered only later. We formalize this mismatch through conditional state update tasks. For causal state update processors, providing the condition first can require exponentially less memory in the worst case than providing it last. Motivated by this principle, we introduce Trace as State. We use collected reasoning traces as a textual proxy for task state and place it before the long-context block on a fresh pass, allowing information derived previously to guide rereading. We conduct extensive experiments on Trace as State and Trace Append, a matched control that uses the same task state proxy but put it after the context. Across three models and three long-context datasets, Trace as State outperforms Trace Append in 26 of 27 reported combinations of model, task, and metric. On GraphWalks Parents, exact match lifts DeepSeek V4 Pro Preview from 29.2% on the initial pass and 43.0% with Trace Appendto 81.8% with Trace as State, and from 66.4% and 83.2% to 100.0% for GLM-5.2. These results show that placing traces before the context can improve long-context reasoning while retaining the causal transformer structure.

Problem

• long-context reasoning
• causal processing
• task state

Innovation

• Trace as State
• long-context reasoning
• causal transformer
• reasoning traces
• conditional state update