One Size Does Not Fit All: Architecture-Aware Adaptive Batch Scheduling with DEBA

2025-11-05

AI Summary

Existing adaptive batch-size methods commonly assume a “one-size-fits-all” strategy across architectures, yet this work reveals substantial architectural heterogeneity in responsiveness to batch-size adaptation. To address this, we propose DEBA—a novel architecture-aware dynamic batch-size scheduling method. DEBA introduces a gradient stability characterization framework grounded in gradient variance, gradient norm dynamics, and loss evolution, augmented by sliding-window statistics and a cooling period mechanism. Crucially, it establishes, for the first time, that batch-size adaptation must be explicitly architecture-specific. Extensive experiments on CIFAR across diverse architectures—including ResNet, DenseNet, and ViT—demonstrate that DEBA accelerates lightweight and moderately deep models by 45–62% with 1–7% accuracy gains; ResNet-18 achieves 36–43% speedup and +2.4–4.0% accuracy; while ViT-B16 yields only 6% speedup—empirically validating both the necessity and efficacy of architecture-aware adaptation.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Adaptive batch size methods aim to accelerate neural network training, but existing approaches apply identical adaptation strategies across all architectures, assuming a one-size-fits-all solution. We introduce DEBA (Dynamic Efficient Batch Adaptation), an adaptive batch scheduler that monitors gradient variance, gradient norm variation and loss variation to guide batch size adaptations. Through systematic evaluation across six architectures (ResNet-18/50, DenseNet-121, EfficientNet-B0, MobileNet-V3, ViT-B16) on CIFAR-10 and CIFAR-100, with five random seeds per configuration, we demonstrate that the architecture fundamentally determines adaptation efficacy. Our findings reveal that: (1) lightweight and medium-depth architectures (MobileNet-V3, DenseNet-121, EfficientNet-B0) achieve a 45-62% training speedup with simultaneous accuracy improvements of 1-7%; (2) shallow residual networks (ResNet-18) show consistent gains of +2.4 - 4.0% in accuracy, 36 - 43% in speedup, while deep residual networks (ResNet-50) exhibit high variance and occasional degradation; (3) already-stable architectures (ViT-B16) show minimal speedup (6%) despite maintaining accuracy, indicating that adaptation benefits vary with baseline optimization characteristics. We introduce a baseline characterization framework using gradient stability metrics (stability score, gradient norm variation) that predicts which architectures will benefit from adaptive scheduling. Our ablation studies reveal critical design choices often overlooked in prior work: sliding window statistics (vs. full history) and sufficient cooldown periods (5+ epochs) between adaptations are essential for success. This work challenges the prevailing assumption that adaptive methods generalize across architectures and provides the first systematic evidence that batch size adaptation requires an architecture-aware design.

Problem

• Existing adaptive batch methods use identical strategies across all neural architectures
• Different network architectures show varying effectiveness with batch size adaptation
• No framework exists to predict which architectures benefit from adaptive scheduling

Innovation

• DEBA monitors gradient variance, norm and loss variations
• It uses sliding window statistics and sufficient cooldown periods
• Architecture-aware design predicts adaptation benefits via gradient stability