Adaptive Data Dropout: Towards Self-Regulated Learning in Deep Neural Networks

2026-04-14

AI Summary

This work addresses the inefficiency of conventional deep neural network training, which uniformly utilizes all training data despite varying sample contributions across different learning stages, thereby limiting both optimization efficiency and generalization. Inspired by self-regulated learning, the paper proposes an adaptive data dropping framework that introduces, for the first time, a dynamic data selection mechanism driven by real-time feedback from training accuracy. Departing from fixed scheduling strategies, this approach continuously adjusts the subset of data used for training during the learning process. A lightweight stochastic update mechanism is employed to balance online exposure between exploration and consolidation phases. Empirical evaluations demonstrate that the proposed framework significantly reduces the number of effective training steps on standard image classification benchmarks while maintaining accuracy comparable to that of static data dropping methods.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Deep neural networks are typically trained by uniformly sampling large datasets across epochs, despite evidence that not all samples contribute equally throughout learning. Recent work shows that progressively reducing the amount of training data can improve efficiency and generalization, but existing methods rely on fixed schedules that do not adapt during training. In this work, we propose Adaptive Data Dropout, a simple framework that dynamically adjusts the subset of training data based on performance feedback. Inspired by self-regulated learning, our approach treats data selection as an adaptive process, increasing or decreasing data exposure in response to changes in training accuracy. We introduce a lightweight stochastic update mechanism that modulates the dropout schedule online, allowing the model to balance exploration and consolidation over time. Experiments on standard image classification benchmarks show that our method reduces effective training steps while maintaining competitive accuracy compared to static data dropout strategies. These results highlight adaptive data selection as a promising direction for efficient and robust training. Code will be released.

Problem

• adaptive data selection
• deep neural networks
• self-regulated learning
• data efficiency
• training dynamics

Innovation

• Adaptive Data Dropout
• self-regulated learning
• dynamic data selection
• training efficiency
• stochastic update mechanism