The Weakest Link Tells It All: Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment

2026-06-26

AI Summary

This work addresses the credit assignment challenge in process reward modeling when training relies solely on the correctness of final answers, where individual reasoning steps receive no explicit supervision. The authors propose a Learnable Credit Assignment (LCA) framework that, for the first time, incorporates the “weakest link” principle into outcome-supervised process reward modeling. They formalize the problem as multiple instance learning and introduce a Softmax-weighted sum pooling mechanism to effectively handle strong dependencies and redundancy among reasoning steps. By jointly optimizing credit assignment and reward modeling, LCA significantly outperforms existing outcome-supervised methods across diverse tasks and large language model backbones, demonstrating enhanced capability in identifying reasoning errors and improving overall performance.

Abstract

Process reward models (PRMs) enhance the reasoning capabilities of large language models (LLMs) by providing fine-grained feedback, yet training PRMs typically requires expensive stepwise annotations. Outcome-supervised PRMs offer a scalable alternative by learning from final-answer correctness alone, but this introduces a fundamental *credit assignment* challenge, i.e., attributing outcomes to responsible reasoning steps. Existing approaches rely on either uniform or causal assignment, both of which fail to anchor credit in step correctness and thus hinder process error identification. In this work, we propose Outcome-Supervised Process Reward Modeling via **L**earnable **C**redit **A**ssignment (**LCA**), an outcome-supervised PRM framework that jointly learns credit assignment and reward modeling under the principle of *Weakest Link Assignment: a reasoning chain is as strong as its weakest link*. To address mutual dependence between credit assignment and reward modeling, we formalize outcome-supervised PRM as a Multiple Instance Learning (MIL) problem and introduce Softmax-Weighted-Sum (SWS) pooling, an MIL pooling technique tailored for strong dependence and redundancy among reasoning states. We prove Bayes consistency of our algorithm under mild assumptions. Extensive experiments demonstrate that **LCA** consistently outperforms state-of-the-art outcome-supervised PRMs across multiple tasks and backbones. Code is available at https://anonymous.4open.science/r/LCA.

Problem

• credit assignment
• process reward modeling
• outcome-supervised learning
• reasoning steps
• weak link

Innovation

• Learnable Credit Assignment
• Outcome-Supervised PRM
• Weakest Link Assignment
• Multiple Instance Learning
• Softmax-Weighted-Sum Pooling