Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learning

2026-07-08

AI Summary

This work addresses the challenge in multi-task reinforcement learning where mismatched exploration-exploitation tempos across tasks induce severe policy entropy fluctuations, undermining training stability and performance. To mitigate this issue, the paper proposes Entropy-Paced Policy Optimization (EPPO), the first method to explicitly model and alleviate the “entropy interference” phenomenon among tasks. Built upon the Group Relative Policy Optimization framework, EPPO replaces fixed clipping thresholds with a task-level dynamic clipping mechanism featuring entropy-aware adaptive clipping bounds, enabling coordinated regulation of exploration tempos across tasks. Experimental results demonstrate that EPPO significantly outperforms existing approaches on standard multi-task benchmarks, effectively suppressing entropy volatility and enhancing both overall performance and training stability.

Abstract

Recent breakthroughs of Reinforcement Learning (RL) have highlighted its potential for complex agentic Large Language Model (LLM) tasks. However, existing efforts largely focus on single-task settings, whereas real-world deployment necessitates a generalist agent capable of solving multiple tasks simultaneously. In this work, we identify a critical yet underexplored phenomenon in multi-task agentic RL: different tasks can exhibit exploration-exploitation pace mismatch. Specifically, easier tasks may converge early to low-entropy policies that hinder learning on harder tasks, while harder tasks can, in turn, push easier tasks back toward high-entropy exploration. This back-and-forth interaction creates inter-task entropy crossovers and frequent entropy spikes. Inspired by this observation, we introduce Entropy Pacing Policy Optimization (EPPO) for multi-task agentic LLMs, which coordinates entropy across tasks to stabilize multi-task optimization. At the core of EPPO is a task-wise dynamic clipping mechanism that replaces the fixed clipping threshold in Group Relative Policy Optimization (GRPO) with a task entropy-aware adaptive bound, tightening updates for over-confident tasks while relaxing them for under-explored ones. Experiments on the multi-task agentic benchmarks demonstrate that the proposed EPPO yields results superior to its counterparts.

Problem

• multi-task reinforcement learning
• exploration-exploitation tradeoff
• entropy dynamics
• agentic LLMs
• policy optimization

Innovation

• Entropy Pacing
• Multi-Task Reinforcement Learning
• Agentic LLMs
• Dynamic Clipping
• Exploration-Exploitation Tradeoff