UNIPO: Unified Interactive Visual Explanation for RL Fine-Tuning Policy Optimization

2026-05-12

AI Summary

Existing reinforcement learning fine-tuning algorithms—such as GRPO and DAPO—suffer from inconsistent design and formulation, hindering effective comparison and comprehension, particularly for non-expert users. To address this challenge, this work proposes the first interactive visualization tool that unifies multiple algorithms under a cohesive interface. By integrating three coordinated views—training overviews, step-level input-output inspection, and side-by-side algorithm comparison—the tool enables token-level tracking of policy optimization dynamics. Built upon frontend visualization technologies and training log parsing, it supports fine-grained representation of mainstream RL fine-tuning algorithms, substantially lowering the barrier to entry for educational demonstrations and algorithm selection. The tool is open-sourced and publicly accessible.

Abstract

Reinforcement learning has emerged as a dominant technique for fine-tuning the behavior of large language models, with policy optimization (PO) algorithms such as GRPO, DAPO, and Dr. GRPO emerging in rapid succession to advance state-of-the-art reasoning and alignment performance. However, the modular differences between these algorithms, including targeted improvements to clipping, advantage estimation, and reward aggregation, are introduced across separate papers with inconsistent notation, making them difficult to compare and intimidating to the non-expert community. We present UNIPO, the first interactive visualization tool that exposes the token-level training dynamics of RL fine-tuning algorithms through a unified design. UNIPO connects three complementary views, a high-level training overview, a step-level prompt and response inspector, and a side-by-side algorithm comparison, allowing learners to observe how individual design decisions propagate through training. Through two usage scenarios, we demonstrate how UNIPO supports both classroom instruction for non-experts and algorithm selection for AI practitioners. Our tool is open-source and publicly available at https://poloclub.github.io/unipo.

Problem

• reinforcement learning
• policy optimization
• large language models
• algorithm comparison
• visual explanation

Innovation

• interactive visualization
• reinforcement learning fine-tuning
• policy optimization
• algorithm comparison
• token-level dynamics