GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training

2026-05-25

AI Summary

This work addresses the limitation of existing hybrid training methods that rely on fixed mixing strategies, which struggle to adapt to the dynamically varying noise present in supervised fine-tuning and reinforcement learning signals. To overcome this, the paper proposes Gradient Adaptive Control (GAC), a method that dynamically adjusts the mixing weights by online estimation of gradient variance and inconsistency between the two signal types. GAC incorporates a noise-aware controller augmented with smoothing mechanisms, prior guidance, and bounded updates, achieving adaptive optimization with negligible computational overhead (<1%). Experimental results demonstrate that GAC consistently outperforms strong baselines across benchmarks in mathematics, code generation, scientific reasoning, and logical tasks, with performance gains becoming more pronounced as model scale increases.

Abstract

Hybrid post-training usually combines supervised fine-tuning and reinforcement learning, but fixed mixing schedules cannot adapt when the relative noise of the two signals changes over time. We propose GAC, a noise-aware controller that derives an adaptive mixing weight from online estimates of gradient variance and disagreement between the two training signals. The method adds smoothing, prior guidance, and bounded updates while reusing existing training tensors. Experiments on math, code, science, and logic benchmarks show that GAC consistently improves hybrid post-training over strong fixed and rule-based baselines, with larger gains at larger model scales and less than 1% training overhead.

Problem

• hybrid post-training
• noise-aware
• adaptive mixing
• gradient variance
• training signal disagreement

Innovation

• adaptive mixing
• noise-aware control
• hybrid post-training
• gradient variance
• online estimation