Memory Constrained Dynamic Subnetwork Update for Transfer Learning

2025-10-23

AI Summary

Edge-device neural network training is severely constrained by limited memory, hindering efficient adaptation to downstream tasks. To address this, we propose MeDyate—a framework for memory-efficient transfer learning under stringent memory budgets. First, we introduce Layer Ranking (LaRa) to quantify parameter importance per layer. Second, we design a dynamic channel resampling strategy guided by the temporal stability of channel importance distributions across epochs, enabling adaptive subnetwork updates. Third, we integrate subnetwork freezing and activation to minimize runtime memory footprint. MeDyate achieves fine-tuning with only hundreds of kB RAM. Evaluated across diverse tasks and architectures, it significantly outperforms existing static and dynamic methods, attaining state-of-the-art accuracy while maintaining ultra-low memory overhead (< 0.5 MB). MeDyate establishes a practical, memory-aware paradigm for on-device transfer learning.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

On-device neural network training faces critical memory constraints that limit the adaptation of pre-trained models to downstream tasks. We present MeDyate, a theoretically-grounded framework for memory-constrained dynamic subnetwork adaptation. Our approach introduces two key innovations: LaRa (Layer Ranking), an improved layer importance metric that enables principled layer pre-selection, and a dynamic channel sampling strategy that exploits the temporal stability of channel importance distributions during fine-tuning. MeDyate dynamically resamples channels between epochs according to importance-weighted probabilities, ensuring comprehensive parameter space exploration while respecting strict memory budgets. Extensive evaluation across a large panel of tasks and architectures demonstrates that MeDyate achieves state-of-the-art performance under extreme memory constraints, consistently outperforming existing static and dynamic approaches while maintaining high computational efficiency. Our method represents a significant step towards enabling efficient on-device learning by demonstrating effective fine-tuning with memory budgets as low as a few hundred kB of RAM.

Problem

• Memory-constrained on-device training limits model adaptation to downstream tasks
• Dynamic subnetwork update enables efficient fine-tuning under strict memory budgets
• Achieves state-of-the-art performance with extremely low memory requirements

Innovation

• Dynamic channel sampling strategy for memory efficiency
• LaRa layer ranking metric enables principled layer selection
• Importance-weighted resampling between training epochs