Entropy-Based Adaptive Weighting for Self-Training

2025-03-31

AI Summary

To address the low data efficiency of large language models (LLMs) in mathematical reasoning self-training, this paper proposes an adaptive weighted self-training method grounded in prediction entropy. The core innovation lies in the first use of prediction entropy as a continuous, tunable basis for weight assignment, implemented via a learnable Sigmoid function to enable fine-grained, dynamic importance modeling—thereby prioritizing learning from high-uncertainty, information-rich samples. Evaluated on GSM8K and MATH benchmarks, the method achieves a ~1% relative improvement on MATH (baseline: 0%) and an additional 1–2% gain on GSM8K over standard self-training. These results demonstrate substantially enhanced generalization capability and training efficiency for challenging mathematical reasoning tasks.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The mathematical problem-solving capabilities of large language models have become a focal point of research, with growing interests in leveraging self-generated reasoning paths as a promising way to refine and enhance these models. These paths capture step-by-step logical processes while requiring only the correct answer for supervision. The self-training method has been shown to be effective in reasoning tasks while eliminating the need for external models and manual annotations. However, optimizing the use of self-generated data for model training remains an open challenge. In this work, we propose Entropy-Based Adaptive Weighting for Self-Training (EAST), an adaptive weighting strategy designed to prioritize uncertain data during self-training. Specifically, EAST employs a mapping function with a tunable parameter that controls the sharpness of the weighting, assigning higher weights to data where the model exhibits greater uncertainty. This approach guides the model to focus on more informative and challenging examples, thereby enhancing its reasoning ability. We evaluate our approach on GSM8K and MATH benchmarks. Empirical results show that, while the vanilla method yields virtually no improvement (0%) on MATH, EAST achieves around a 1% gain over backbone model. On GSM8K, EAST attains a further 1-2% performance boost compared to the vanilla method.

Problem

• Optimizing self-generated data usage in model training
• Prioritizing uncertain data via adaptive weighting strategy
• Enhancing reasoning ability with entropy-based weighting

Innovation

• Entropy-based adaptive weighting strategy
• Prioritizes uncertain data in self-training
• Enhances reasoning with tunable sharpness parameter