🤖 AI Summary
研究解决了结合大语言模型与强化学习时奖励信号理论地位不明确的问题，通过将LLM反馈作为有界势函数，确保即使在LLM评分不准确的情况下也能保持最优策略集。
📝 Abstract
Combining large language models with reinforcement learning is increasingly explored, yet the theoretical status of LLM-derived reward signals is often left implicit. We formalize the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and show that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate. This guarantee is stronger than what general LLM-as-reward approaches provide. We verify the result numerically on a small MDP under four potential configurations, including an adversarial one scaled to twenty times the base reward magnitude.