Process Reward Models for LLM Agents: Practical Framework and Directions

2025-02-14

AI Summary

This work addresses the challenge of continual interactive learning for LLM-based agents without explicit outcome supervision, focusing on three core issues: process reward modeling, exploration efficiency, and inference consistency. To this end, we propose AgentPRM—a framework featuring a lightweight Actor-Critic architecture augmented with Monte Carlo backtracking to enable online-updatable process reward models (PRMs). We further introduce InversePRM, the first method to infer process rewards directly from demonstrations via inverse reinforcement learning—eliminating the need for outcome annotations. This constitutes the first systematic paradigm for process reward modeling tailored to LLM agents, simultaneously advancing reward shaping, exploration guidance, and prediction-reasoning alignment. Evaluated on ALFWorld, our 3B-parameter model substantially outperforms the GPT-4o baseline, demonstrating strong test-time scalability and robustness against reward hacking. The complete implementation is open-sourced.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

We introduce Agent Process Reward Models (AgentPRM), a simple and scalable framework for training LLM agents to continually improve through interactions. AgentPRM follows a lightweight actor-critic paradigm, using Monte Carlo rollouts to compute reward targets and optimize policies. It requires minimal modifications to existing RLHF pipelines, making it easy to integrate at scale. Beyond AgentPRM, we propose InversePRM, which learns process rewards directly from demonstrations without explicit outcome supervision. We also explore key challenges and opportunities, including exploration, process reward shaping, and model-predictive reasoning. We evaluate on ALFWorld benchmark, show that small 3B models trained with AgentPRM and InversePRM outperform strong GPT-4o baselines, and analyze test-time scaling, reward hacking, and more. Our code is available at: https://github.com/sanjibanc/agent_prm.

Problem

• Train LLM agents for continual improvement
• Integrate with minimal RLHF pipeline modifications
• Learn process rewards without outcome supervision

Innovation

• AgentPRM framework for LLM agents
• Monte Carlo rollouts for reward targets
• InversePRM learns from demonstrations