CAPO: Counterfactual Credit Assignment in Sequential Cooperative Teams

2026-04-19

AI Summary

This work addresses the challenge of credit assignment in sequential cooperative teams, where agents share a global reward and update policies sequentially, making individual contributions difficult to evaluate—especially when historical data becomes inconsistent with the current policy. The paper introduces Sequential Aristocrat Utility (SeqAU), the first extension of Wolpert and Tumer’s Aristocrat Utility framework to sequential settings, providing each agent with an individualized signal that maximizes action learnability and yielding a closed-form counterfactual advantage estimator. Building on SeqAU, the authors propose CAPO, a critic-free policy gradient algorithm that efficiently decomposes team rewards and computes advantages using only an initial batch of data and minimal forward passes under the current policy. Experiments demonstrate that CAPO significantly outperforms baselines in sequential multi-armed bandit tasks, with performance gains amplifying as team size increases, highlighting its potential for real-world applications such as multi-LLM pipelines.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Multiagent Systems: Teamwork
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In cooperative teams where agents act in a fixed order and share a single team reward, it is hard to know how much each agent contributed, and harder still when agents are updated one at a time because data collected earlier no longer reflects the new policies. We introduce the Sequential Aristocrat Utility (SeqAU), the unique per-agent learning signal that maximizes the individual learnability of each agent's action, extending the classical framework of Wolpert and Tumer (2002) to this sequential setting. From SeqAU we derive CAPO (Counterfactual Advantage Policy Optimization), a critic-free policy-gradient algorithm. CAPO fits a per-agent reward decomposition from group rewards and computes the per-agent advantage in closed form plus a handful of forward passes through the current policy, requiring no extra environment calls beyond the initial batch. We give analytic bias and variance bounds and validate them on a controlled sequential bandit, where CAPO's advantage over standard baselines grows with the team size. The framework is general; multi-LLM pipelines are a natural deployment target.

Problem

• credit assignment
• sequential cooperative teams
• counterfactual reasoning
• multi-agent reinforcement learning
• reward decomposition

Innovation

• Counterfactual Credit Assignment
• Sequential Aristocrat Utility
• Critic-Free Policy Gradient
• Reward Decomposition
• Multi-Agent Cooperation