🤖 AI Summary
When models are updated using trajectory-based losses yet evaluated by task rewards, the update direction often misaligns with decision utility. This work proposes a theoretical framework for trajectory-level transfer, introducing the notions of learnability and decision utility to analyze how single-step and multi-step trajectory updates can jointly reduce surrogate loss and decision risk. By leveraging gradient alignment, second-order curvature modeling, path-dependent accumulation bounds, and refined candidate divergence analysis, the study elucidates how gradient alignment, calibration error, and curvature jointly influence transfer efficacy, and establishes a trade-off between approximation accuracy and calibration. The theoretical findings are validated through experiments in grid-world environments and post-training of large language models, supporting the effectiveness of the proposed transfer conditions and selection strategies.
📝 Abstract
A broad range of models face the mismatch where they are updated through trajectory losses but are evaluated by downstream task reward. Here, a trajectory is a training instance that induces a surrogate loss whose reduction might not track the model's decision utility update. Theoretically, we ask when one step of trajectory training reduces both population surrogate loss and decision risk, and how transfer accumulates along repeated updates. To formalize this, we first fix a checkpoint and a restricted update space, and define the reductions in population surrogate risk and decision risk induced by a trajectory as its learnability and decision utility, respectively. On this basis, our theory yields four main results. First, a one-step transfer bound separates their discrepancy into first-order gradient misalignment after nonnegative calibration and second-order curvature; and a pathwise extension accumulates the same terms over repeated updates. Second, when the accessible surrogate gradient is nonzero, universal first-order transfer over every accessible direction holds exactly when the accessible surrogate and decision gradients are positively collinear. Third, the calibration gap bounds the decision regret of learnability-based trajectory selection, while a candidate-difference refinement tightens this guarantee by retaining only directions that affect pairwise rankings. Finally, we establish an approximation--calibration trade-off across nested update spaces. Controlled gridworld and LLM post-training experiments yield results consistent with our predictions.