What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents

2026-05-19

AI Summary

This work addresses the challenge of long-term credit assignment for multi-turn agents operating under sparse reward settings. To this end, the authors propose the SERL framework, which integrates task-level rewards with fine-grained environmental feedback—such as error messages and page changes—to enable precise optimization of critical actions. SERL presents the first systematic evaluation of five categories of environmental feedback combined with two insertion granularities and introduces a novel selective post-hoc distillation mechanism that leverages synergies between task rewards and environmental signals. Experimental results demonstrate that SERL achieves state-of-the-art performance, attaining success rates of 90.0% on ALFWorld and 80.1% on WebShop, significantly outperforming existing reinforcement learning and distillation approaches.

Abstract

Reinforcement learning can train LLM agents from sparse task rewards, but long-horizon credit assignment remains challenging: a single success-or-failure signal must be distributed across many actions. Existing methods rely on trajectory-level rewards or proxy signals, without fully leveraging per-step environmental feedback. Multi-turn agent settings are underexplored, where feedback can include error messages, page changes, observations, or reference trajectories. We systematically study five feedback sources and two insertion granularities and introduce SERL, a selective environment-reweighted learning framework. SERL uses the task reward to determine update direction, while environment feedback adjusts placement and magnitude, focusing on critical actions. On ALFWorld and WebShop, SERL achieves 90.0% and 80.1% success, outperforming strong RL and distillation baselines. Analysis shows that grounded, action-relevant feedback at meaningful points consistently outperforms indiscriminate use of longer or richer context.

Problem

• credit assignment
• multi-turn agents
• environment feedback
• reinforcement learning
• sparse rewards

Innovation

• selective distillation
• environment feedback
• multi-turn agents
• credit assignment
• reinforcement learning