Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs

2026-05-27

AI Summary

This study investigates how sample difficulty influences the multi-step reasoning capabilities of large language models trained with reinforcement learning from verifiable rewards (RLVR). By employing difficulty-stratified sampling and single-example analysis, complemented with Temporal Sparse Autoencoders (T-SAE) to probe internal feature dynamics, the work reveals—for the first time—a non-monotonic relationship between RLVR performance and sample difficulty: moderately difficult samples most effectively enhance reasoning ability, whereas overly challenging samples often induce degenerate behaviors. Building on this insight, the authors propose a difficulty-adaptive training strategy that integrates backward-reasoning reconstruction with T-SAE-derived guidance signals, substantially mitigating reward sparsity and credit assignment issues, thereby improving both the stability and final performance of RLVR training.

Abstract

Reinforcement Learning with Verifiable Reward (RLVR) is empirically shown to notably enhance the reasoning performance of large language models (LLMs), particularly in mathematics and programming. However, the mechanistic role of Sample Difficulty in RLVR remains poorly understood. In this paper, we investigate RLVR through the lens of difficulty-wise and one-sample analysis. We find that sample difficulty has a non-monotonic effect on RLVR: easy and medium-difficulty problems yield the strongest and most stable reasoning improvements, whereas overly hard problems often provide weak learning signals, induce degenerate behaviors such as answer repetition or skipping necessary computation, and can ultimately degrade the model's pre-existing capabilities. Beyond the obverse of response, we further analyze the model's internal feature dynamics using Temporal Sparse Autoencoders (T-SAE). Easy problems mainly reinforce direct-answer and basic-computation features while suppressing deliberative-reasoning features; hard problems activate reasoning-related features but become useful only when successful trajectories are sampled; medium-difficulty problems provide a more balanced signal, strengthening both computation and multi-step reasoning features. Motivated by these findings, we propose difficulty-adaptive strategies for hard-sample utilization, using backward-reasoning reformulation and T-SAE-guided training signals to improve reward density and credit assignment during RLVR. Overall, our results identify sample difficulty as a key factor governing both the optimization dynamics and representation evolution of RLVR.

Problem

• Sample Difficulty
• Reinforcement Learning with Verifiable Reward
• Large Language Models
• Reasoning Performance
• Representation Evolution

Innovation

• Sample Difficulty
• Reinforcement Learning with Verifiable Reward (RLVR)
• Temporal Sparse Autoencoders (T-SAE)
• difficulty-adaptive training
• feature dynamics