Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrations

2026-07-05

AI Summary

This work addresses the challenge of inefficient learning in large language model (LLM) agents operating in partially observable environments with long-horizon tasks, where effective supervision signals are often scarce. The authors propose Hindsight Supervised Learning (HSL), a novel framework that systematically leverages implicit successful outcomes embedded within agent trajectories. By retrospectively relabeling trajectories with achieved goals—converting unintended but realized outcomes into supervisory signals—and combining this with irrelevant-action masking and sample reweighting, HSL enables more effective fine-tuning without requiring additional human annotations. The method seamlessly integrates with standard supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) pipelines. Evaluated on benchmarks such as ALFWorld, HSL achieves substantial performance gains over baseline models trained on full demonstration datasets, using only one-quarter of the original expert data.

Abstract

Large language model agents operate in partially observable, long-horizon settings where obtaining supervision remains a major bottleneck. We address this by utilizing a source of supervision overlooked in existing post-training methods: unintended yet successful goals embedded within agent rollouts. Specifically, we introduce Hindsight Supervised Learning (HSL), where an auxiliary LLM reviews each completed trajectory and relabels it with all of the natural-language goals the agent actually achieved. HSL then pairs the trajectory with its relabeled goals and uses these pairs for additional fine-tuning. To mitigate suboptimality in the relabeled data, we propose two learning techniques for HSL, irrelevant-action masking and sample reweighting. Our experiments show that HSL is flexible and compatible with existing post-training pipelines. It improves both SFT and DPO, with larger gains on long-horizon tasks with more diverse goal spaces. Moreover, HSL is sample-efficient: on ALFWorld, it surpasses baselines trained on the full dataset while using only one quarter of the ground-truth demonstrations.

Problem

• supervision bottleneck
• long-horizon tasks
• partially observable environments
• demonstration learning
• goal relabeling

Innovation

• Hindsight Supervised Learning
• trajectory relabeling
• goal relabeling
• sample-efficient fine-tuning
• LLM agents