Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents

2026-06-24

AI Summary

This work addresses the challenge in population-based reinforcement learning where semantically similar intermediate steps receive conflicting credit due to differing trajectory outcomes, leading to gradient conflicts and the underutilization of valid progress in failed trajectories. To resolve this, the authors propose a value-free reward shaping method that introduces, for the first time, a semantic consistency–based credit assignment mechanism. By aligning failed trajectories with successful ones from the same population based on semantic similarity, the approach assigns positive step-level rewards to consistent novel progress without relying on value function estimates, thereby enabling cross-trajectory credit transfer. Evaluated on ALFWorld and WebShop, the method achieves success rates of 93.7 ± 4.1% and 74.8 ± 2.0%, respectively, using a 1.5B-parameter model, significantly enhancing sample efficiency in multi-step task learning.

Abstract

Group-based reinforcement learning effectively post-trains LLM agents for long-horizon, sparse-reward tasks by deriving step-level credit from trajectory outcomes. However, this ties a step's credit to its rollout's final outcome: semantically near-identical intermediate steps receive opposite credit depending on whether their trajectory eventually succeeded or failed. Such semantic credit inconsistency sends conflicting gradients to similar actions and wastes the partially-correct progress inside failed rollouts. Motivated by this, we propose Semantic Consistency Policy Optimization (SCPO), a value-free reward-shaping method that mitigates this inconsistency by recovering step-level credit from successful siblings in the same rollout group. Concretely, SCPO scores each failed step against a successful sibling and adds positive step-level credit for new progress along that sibling. On ALFWorld and WebShop, SCPO matches or exceeds strong group-based baselines, reaching 93.7+/-4.1 percent success on ALFWorld and 74.8+/-2.0 percent on WebShop at 1.5B parameters, with gains concentrated on the hardest multi-step tasks.

Problem

• semantic consistency
• credit assignment
• reinforcement learning
• LLM agents
• sparse-reward tasks

Innovation

• Semantic Consistency
• Reward Shaping
• Reinforcement Learning
• LLM Agents
• Credit Assignment