ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy

2026-07-03

AI Summary

This work addresses the token-level credit assignment challenge in large language models trained with reinforcement learning under sparse rewards. To this end, the authors propose an adaptive credit assignment framework based on fine-grained surrogate entropy. The method employs asymmetric policy modulation—enhancing exploration of uncertain tokens in successful trajectories while suppressing overconfident tokens in failed ones—to achieve precise credit allocation. Additionally, it incorporates modality alignment constraints and proximal policy updates to stabilize gradient directions and mitigate non-local interference. Experimental results demonstrate that the proposed approach significantly outperforms strong baselines such as DAPO, GTPO, and SAPO on mathematical and code generation benchmarks, including AIME 2025 and HumanEvalPro.

Abstract

Reinforcement Learning (RL) has substantially improved the reasoning ability of large language models (LLMs), but sparse outcome rewards still make token-level credit assignment difficult. Existing scalable RL methods typically assign trajectory-level rewards uniformly across tokens, while recent entropy-aware approaches either rely on coarse detached heuristics or directly optimize true entropy, which can introduce non-local gradient components misaligned with sampled-token policy updates. We propose Adaptive Credit Policy Optimization (ACPO), a token-level credit assignment framework based on a mode-local surrogate entropy. ACPO asymmetrically modulates policy updates by emphasizing uncertain decisions in successful rollouts and overconfident tokens in failed rollouts. We show that the surrogate admits deterministic entropy bounds and, under modal alignment and proximal updates, preserves the policy-gradient direction to leading order. Experiments on mathematical reasoning and coding benchmarks, including AIME 2025 and HumanEvalPro, show that ACPO consistently improves over strong RL baselines such as DAPO, GTPO, and SAPO.

Problem

• credit assignment
• reinforcement learning
• large language models
• token-level reward
• entropy optimization

Innovation

• Adaptive Credit Assignment
• Surrogate Entropy
• Token-level Policy Optimization
• Reinforcement Learning for LLMs
• Mode-local Entropy