Dynamic Important Example Mining for Reinforcement Finetuning

2026-08-29

AI Summary

本文提出动态重要样本挖掘(DIEM)方法，通过自适应选择训练数据以优化强化微调过程中的策略学习效果。

Abstract

Reinforcement fine-tuning (RFT) is increasingly used to strengthen the reasoning abilities of large models, yet its effectiveness is bound by how training data are selected and used. Most data-centric RFT methods rely on static or heuristic sample selection, implicitly assuming a sample's value is fixed over training. This overlooks the non-stationary dynamics of policy learning and can lead to suboptimal updates. We propose Dynamic Important Example Mining (DIEM), a principled and fully automated framework that makes data utilization adaptive throughout RFT. DIEM integrates two components into each optimization step: (i) a gradient-alignment importance estimator that efficiently approximates each sample's marginal contribution to policy improvement; and (ii) a constrained batch reweighting scheme that maximizes aggregate utility while preserving the update's gradient magnitude to stabilize optimization. Across several reasoning benchmarks, DIEM consistently outperforms strong static and dynamic baselines. The code will be released via https://github.com/hrtan/DIEM.

Problem

• Reinforcement fine-tuning
• data selection
• non-stationary dynamics

Innovation

• Dynamic Important Example Mining
• Reinforcement Fine-tuning
• Gradient-alignment Importance Estimator
• Constrained Batch Reweighting