Tracking the Moving Frontier: Long-Short Term Advantage Estimator

2026-09-06

AI Summary

本文提出长短期优势估计器LSTAE，通过利用历史经验来减少重复采样，有效降低了长期代理训练成本。

Abstract

Group-based RLVR methods estimate advantages by repeatedly sampling multiple trajectories for each prompt, making long-horizon agent training expensive and discarding useful experience accumulated across iterations. We ask whether historical experience can replace these repeated within-iteration comparisons without directly optimizing on stale trajectories. We introduce Long-Short Term Advantage Estimator (LSTAE), a single-stream RL algorithm that uses history for advantage estimation while updating the policy only with the current rollout. LSTAE maintains a persistent tracker for each task anchor. At the trajectory level (long term), a drift-aware historical baseline tracks the anchor's moving success frontier and measures the relative contribution of each new trajectory. At the step level (short term), a recent state-experience buffer exploits recurrent states to estimate localized action advantages. This two-timescale design converts accumulated experience into multi-granular credit signals, requiring only one rollout per anchor. Across agentic and mathematical reasoning benchmarks, LSTAE matches or improves upon strong group-based baselines while substantially reducing rollout cost.

Problem

• Long-horizon Agent Training
• Historical Experience
• Advantage Estimation
• Group-based RLVR Methods
• Trajectory Sampling

Innovation

• Long-Short Term Advantage Estimator
• historical experience
• drift-aware historical baseline
• recent state-experience buffer
• multi-granular credit signals