TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing

2026-05-11

AI Summary

This work addresses a critical yet previously unexamined issue in multi-task reinforcement learning: the pathological conditioning of value network gradients in Proximal Policy Optimization (PPO), which causes learning stagnation on tail tasks and dominance by easier tasks during policy updates. To mitigate this, the authors propose Critic Balancing—a lightweight mechanism that restructures PPO through gradient normalization and task-aware value target design, thereby improving gradient conditioning and balancing learning dynamics across tasks without resorting to modular architectures or large models. Evaluated on the Meta-World+ benchmark, the method significantly outperforms strong off-policy baselines such as SAC and ARS, achieving superior performance with fewer parameters and environment interaction steps—particularly excelling on tail tasks and during early training stages—and establishes a new state of the art for on-policy methods in multi-task settings.

Abstract

Soft Actor-Critic (SAC) and its variants dominate Multi-Task Reinforcement Learning (MTRL) due to their off-policy sample efficiency, while on-policy methods such as Proximal Policy Optimization (PPO) remain underexplored. We diagnose that PPO in MTRL suffers from a previously overlooked issue: critic-side gradient ill-conditioning, which may cause tail tasks to stall while easy tasks dominate the value function's updates. To address this, we propose TOPPO (Tail-Optimized PPO), a reformulation of PPO via Critic Balancing -- a set of modules that improve gradient conditioning and balance learning dynamics across tasks. Unlike prior approaches that rely on modular architectures or large models, TOPPO targets the optimization bottleneck within PPO itself. Empirically, TOPPO achieves stronger mean and tail-task performance than published SAC-family and ARS-family baselines while using substantially fewer parameters and environment steps on Meta-World+ benchmark. Notably, TOPPO matches or surpasses strong SAC baselines early in training and maintains superior performance at full budget. Ablations confirm the effectiveness of each module in TOPPO and provide insights into their interactions. Our results demonstrate that, with proper optimization, on-policy methods can rival or exceed off-policy approaches in MTRL, challenging the prevailing reliance on SAC and highlighting critic-side gradient conditioning as the central bottleneck.

Problem

• Multi-Task Reinforcement Learning
• Proximal Policy Optimization
• Critic Gradient Conditioning
• Tail Tasks
• Value Function Update

Innovation

• Critic Balancing
• Multi-Task Reinforcement Learning
• Proximal Policy Optimization
• Gradient Conditioning
• Tail-Task Optimization