VERPO: Verified Evidence Regularized Policy Optimization

2026-09-05

AI Summary

为解决语言模型后训练中序列级优势无法指导具体决策的问题，VERPO通过引入验证证据调节策略优化方法，提高任务成功率。

Abstract

Verifiable outcome rewards guide language-model post-training, but sequence-level advantages do not identify which token-level decisions should be preserved or revised. Evidence-conditioned Teachers provide denser supervision by replaying sampled trajectories with privileged feedback. Yet indiscriminate imitation risks transferring formatting or reasoning-style shifts that do not support task success. We introduce VERPO, a Verified Evidence Regularized Policy Optimization framework that treats evidence as a proposal for policy correction while retaining the outcome objective. It separates evidence-free reference restoration from signed token-level evidence corrections. Fisher Evidence Contrast attenuates corrections along an estimated evidence-presence direction. A stopped token-wise ZPD controller scales acceptance according to local reward alignment and Fisher movement cost, while the reference channel remains independent of acceptance. Across five scientific-reasoning and tool-use tasks, the best variant on each backbone exceeds the strongest compared baseline in average score. The averages rise from 0.6826 to 0.6857 on Qwen3-4B, from 0.6895 to 0.7058 on Qwen3-8B, and from 0.4751 to 0.5657 on Llama-3.2-1B.

Problem

• Evidence-conditioned Teachers
• Token-level Decisions
• Policy Optimization

Innovation

• VERPO
• Evidence Correction
• Fisher Evidence Contrast
• Token-wise ZPD Controller