HISPO: Hierarchical Importance-Sampling Policy Optimization with Entropy-Derived Segments

2026-09-14

AI Summary

本文提出HISPO方法，通过构建基于熵的连续段落并应用重要性采样校正来解决长形式数学推理中的信用分配问题，提高了模型在多个基准测试上的性能。

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a central approach for improving mathematical reasoning in language models, but long-form completions introduce a difficult credit-assignment problem: different parts of a solution trace may contribute unevenly to final correctness. Existing policyoptimization objectives for RLVR commonly apply importance-sampling correction at either the token level (GRPO, DAPO) or the sequence level (GSPO), imposing different granularities for assigning credit across a response. We introduce Hierarchical Importance-Sampling Policy Optimization (HISPO), a segment-level policy-optimization method that constructs rollout-time entropy-derived contiguous segments, assigns soft entropy-based saliency weights, and applies clipped importance-sampling correction at the segment granularity. This provides an intermediate correction unit between token-level GRPO/DAPO and sequence-level GSPO. We evaluate HISPO by fine-tuning Qwen3-1.7B-Base on mathematical reasoning tasks. Across six benchmarks, HISPO improves Pass@8 over the strongest baseline on all benchmarks and matches or exceeds the strongest baseline in Acc@8 on five of them. On AIME25, HISPO improves over GRPO by +3.75 Acc@8 and +3.78 Pass@8, and over GSPO by +2.50 Acc@8 and +1.27 Pass@8. These results suggest that segment-level correction is a promising granularity for RLVR in long-form mathematical reasoning.

Problem

• Reinforcement Learning
• Credit Assignment
• Mathematical Reasoning

Innovation

• Hierarchical Importance-Sampling
• Entropy-Derived Segments
• Policy Optimization
• RLVR
• Mathematical Reasoning