SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewards

2026-02-24

AI Summary

This work addresses the challenge that large language models (LLMs) acting as multi-step decision-making agents struggle to effectively leverage their intrinsic uncertainty to guide reinforcement learning. The authors propose an uncertainty-aware reward mechanism that, for the first time, integrates entropy, least confidence, and margin-based metrics to construct token-level uncertainty estimates. Coupled with a failure-aware reward shaping strategy, this approach enables more efficient exploration and policy optimization. Evaluated on the ALFWorld and WebShop benchmarks, the method significantly outperforms strong baselines. Ablation studies further confirm that the introduced uncertainty signals play a critical role in enhancing both exploration efficiency and overall robustness.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Multiagent Systems: Multiagent Systems under Uncertainty
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) are increasingly deployed as multi-step decision-making agents, where effective reward design is essential for guiding learning. Although recent work explores various forms of reward shaping and step-level credit assignment, a key signal remains largely overlooked: the intrinsic uncertainty of LLMs. Uncertainty reflects model confidence, reveals where exploration is needed, and offers valuable learning cues even in failed trajectories. We introduce SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewards, a reinforcement learning framework that incorporates uncertainty directly into the reward design. SELAUR integrates entropy-, least-confidence-, and margin-based metrics into a combined token-level uncertainty estimate, providing dense confidence-aligned supervision, and employs a failure-aware reward reshaping mechanism that injects these uncertainty signals into step- and trajectory-level rewards to improve exploration efficiency and learning stability. Experiments on two benchmarks, ALFWorld and WebShop, show that our method consistently improves success rates over strong baselines. Ablation studies further demonstrate how uncertainty signals enhance exploration and robustness.

Problem

• large language models
• uncertainty
• reward design
• multi-step decision-making
• reinforcement learning

Innovation

• uncertainty-aware rewards
• LLM agent
• reinforcement learning
• reward shaping
• token-level uncertainty