Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought

2026-03-24

AI Summary

This work addresses the limitation of existing reinforcement learning approaches in multimodal chain-of-thought reasoning, which rely on coarse-grained optimization and fail to differentiate reasoning steps based on their degree of visual grounding. To overcome this, we propose the Perception-Exploration Policy Optimization (PEPO) method, which introduces— for the first time—token-level perceptual priors and a smoothing gating mechanism to enable fine-grained policy updates without requiring additional supervision or auxiliary branches. PEPO constructs perceptual priors from hidden state similarities and combines token entropy to derive token-level advantage functions, making it compatible with mainstream RLVR frameworks such as GRPO and DAPO. Extensive experiments across multiple benchmarks—including geometric reasoning, visual grounding, visual puzzles, and few-shot classification—demonstrate that PEPO consistently yields significant performance gains while ensuring more stable training dynamics.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Search and Optimization: Learning to Search
• Machine Learning: Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Multimodal Chain-of-Thought (CoT) reasoning requires large vision-language models to construct reasoning trajectories that interleave perceptual grounding with multi-step inference. However, existing Reinforcement Learning with Verifiable Rewards (RLVR) methods typically optimize reasoning at a coarse granularity, treating CoT uniformly without distinguishing their varying degrees of visual grounding. In this work, we conduct a token-level analysis of multimodal reasoning trajectories and show that successful reasoning is characterized by structured token dynamics reflecting both perceptual grounding and exploratory inference. Building upon this analysis, we propose Perception-Exploration Policy Optimization (PEPO), which derives a perception prior from hidden state similarity and integrates it with token entropy through a smooth gating mechanism to produce token-level advantages. PEPO integrates seamlessly with existing RLVR frameworks such as GRPO and DAPO, requiring neither additional supervision nor auxiliary branches. Extensive experiments across diverse multimodal benchmarks demonstrate consistent and robust improvements over strong RL baselines, spanning geometry reasoning, visual grounding, visual puzzle solving, and few-shot classification, while maintaining stable training dynamics. Code: https://github.com/xzxxntxdy/PEPO

Problem

• Multimodal Chain-of-Thought
• token-level optimization
• perceptual grounding
• Reinforcement Learning with Verifiable Rewards
• vision-language models

Innovation

• Token-Level Policy Optimization
• Multimodal Chain-of-Thought
• Perception-Exploration Policy Optimization
• Perceptual Grounding
• Reinforcement Learning with Verifiable Rewards