A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism

2026-07-14

AI Summary

This study investigates whether Group Relative Policy Optimization (GRPO) can yield meaningful performance improvements for 4B–8B scale language and vision-language web agents. Through systematic control of learning rate, KL penalty weight, and random seeds, the authors evaluate GRPO’s impact relative to a strong supervised baseline across 18 experimental configurations. The work uncovers, for the first time, a learning-rate-gated failure mechanism of GRPO in small models, revealing a double dissociation between performance degradation and collapse. It further demonstrates that the coupling between effective rank and capability is model-scale dependent. Results indicate that GRPO provides no significant gains on already-mastered tasks and can even degrade text-generation performance at medium-to-high learning rates. GRPO proves beneficial only when the sampling policy outperforms the greedy policy, with 4B and 8B models exhibiting markedly distinct internal dynamics.

Abstract

Reinforcement learning with verifiable rewards, and Group Relative Policy Optimization (GRPO) in particular, is now run routinely on a supervised checkpoint in the hope of producing a stronger agent. We ask whether it adds skill to a small language and vision-language model web agent at the 4B to 8B scale, or whether it mostly reshapes behavior the supervised model already has. Across a control grid of 18 runs that varies learning rate, KL weight, seed, initialization, and clipping, no configuration credibly improves the success rate of a strong supervised baseline on tasks the agent has largely mastered. On the text track, moderate to high learning rates make it credibly worse. The null holds under paired testing, 25 evaluation seeds, 6 training seeds, changes to the recipe, both text and Set-of-Marks screenshot observations, and scaling the backbone to 8B; the credible harm is a text-track finding and is only nominal under Set-of-Marks. To show that the null reflects the setting and not a broken pipeline, we run the identical harness, reward, and recipe on tasks whose reward is reachable by sampling, and there the success rate rises by 22 points with a paired interval that excludes zero. GRPO therefore helps only when there is headroom to climb, meaning the sampled policy already succeeds more often than the greedy one. We then explain the failure. A middle learning rate degrades the agent and a high one collapses it, and the two regimes form a double dissociation: grafting localizes the degrade regime to the attention and MLP blocks, while the collapse regime cannot be traced to any single group, and the embedding change that dominates the weight movement is causally inert. At 4B, effective rank in the late layers tracks capability in both directions; at 8B the two come apart. This coupling is specific to the smaller model, so we report it as scale-dependent.

Problem

• Reinforcement Learning
• Group Relative Policy Optimization
• Language Models
• Vision-Language Models
• Learning Rate

Innovation

• Group Relative Policy Optimization
• learning rate gating
• scale-dependent failure
• effective rank
• reinforcement learning null result