deep learning

High barriers to adopting pre-trained models and a lack of empirical guidance for strategy selection hinder practical deployment in few-shot image classification and object detection. Method: We systematically compare linear probing versus fine-tuning across ResNet, MobileNet, and EfficientNet, and propose an end-to-end TensorFlow framework integrating multi-scale feature-space visualization (PCA, t-SNE, UMAP) to unify analysis of representation evolution. Contribution/Results: Linear probing significantly outperforms fine-tuning under extreme data scarcity (≤100 samples per class) while accelerating training by 3–5×. The framework enables high-accuracy, rapid deployment (<1 hour for fine-tuning) on standard benchmarks (ImageNet-1K, CIFAR-100), balancing beginner-friendly usability with expert-level extensibility. It bridges the gap between theoretical representation analysis and real-world engineering practice.

To address inefficient backpropagation and excessive memory overhead in small-scale computational graph training on single-CPU nodes, this paper proposes BurTorch, a lightweight training framework. Methodologically, BurTorch abandons the abstraction-layer bloat of general-purpose deep learning frameworks and instead embraces the fundamentals of compiled-language programming and numerical optimization: it employs hand-optimized C++, static computational graphs, explicit memory management, and a native backpropagation engine grounded in the Linnainmaa/Rumelhart chain rule. This system-level design directly targets performance bottlenecks inherent to small-graph training. Experiments demonstrate that, on representative small-graph tasks, BurTorch achieves up to 2000× speedup and 3500× memory reduction over PyTorch; even on a micro-scale GPT-3 model, it delivers 20× acceleration and 80× memory savings. These results significantly advance efficient, resource-constrained deep learning training.

Existing hardware-software co-design tools struggle to accurately model memory consumption and backward-pass complexity in neural network training. This work proposes the first extension of the experimentally validated inference modeling framework, Stream, to the training domain, introducing a comprehensive framework for modeling and optimizing training on heterogeneous dataflow accelerators. The framework supports training workflow modeling, exploration of layer fusion configurations, and optimization of activation checkpointing strategies. Integrated with a genetic algorithm for hardware architecture search, it is validated on ResNet-18 and a small-scale GPT-2 model, effectively uncovering critical trade-offs between performance and memory in training-specific hardware design and identifying superior architectures and training strategies.

This study addresses the fundamental question: “Can purely algorithmic improvements yield practical acceleration in neural network training?” To this end, we organized the inaugural AlgoPerf competition, establishing— for the first time—two rigorous evaluation paradigms: workload-agnostic assessment and hyperparameter-free benchmarking, with end-to-end training time on identical hardware as the sole primary metric. Methodologically, we developed a multi-task benchmarking framework integrating Distributed Shampoo (a non-diagonal preconditioner) and Schedule-Free AdamW (a hyperparameter-free optimizer), complemented by standardized temporal measurement protocols and fairness-preserving engineering safeguards. Results show that Distributed Shampoo achieved top performance in the hyperparameter-tuned track, while Schedule-Free AdamW led in the hyperparameter-free track. Top-performing methods demonstrated consistent speedups across diverse CV and NLP tasks, empirically validating that high-quality algorithmic design delivers substantial and robust training acceleration.

Modern large-scale distributed training faces sharply diminishing returns in hardware scaling: as GPU counts reach thousands, communication overhead dominates performance bottlenecks, rendering conventional parallelism strategies—data, tensor, and pipeline parallelism—suboptimal. Method: Leveraging real-world LLM training workloads, this project establishes an empirical analytical framework spanning diverse model scales, hardware configurations, and parallelization strategies. It quantifies the nonlinear relationship between accelerator count and performance gain, precisely identifying critical inflection points across model, data, and compute scaling dimensions. Contribution/Results: We discover that low-communication “suboptimal” strategies become optimal at extreme scale; we empirically determine hardware selection criteria, cluster topology requirements, and optimal parallelism combinations for training billion-parameter models. Our findings provide actionable, deployment-ready optimization guidelines for trillion-parameter LLM training infrastructures.

This study addresses the inequities in AI accessibility arising from imbalanced computational resource allocation, high energy consumption, and hardware barriers in large-scale deep learning training. Conducting systematic benchmarking on Intel Xeon CPUs and NVIDIA Tesla T4 GPUs across TensorFlow and PyTorch frameworks, we evaluate four representative models—Conv6, VGG16, ResNet18, and CycleGAN. For the first time, we jointly analyze model complexity, framework-level optimizations, and hardware platforms to quantify GPU acceleration benefits, revealing that lightweight models like Conv6 achieve the most significant speedup (up to 246× in training). We also find that TensorFlow’s operator fusion reduces inference latency by approximately 15%. Using polynomial regression, we project GPU memory requirements through 2025 and argue that shared GPU infrastructure is a critical pathway toward democratizing AI and ensuring equitable research opportunities.

This work addresses the challenge of defect detection in deep learning libraries such as TensorFlow and PyTorch, where complex APIs often lead to subtle bugs and existing testing approaches suffer from high false-positive rates due to imprecise specifications. To overcome this limitation, the authors propose a machine learning classifier that leverages tensor shape abstraction as a precise input representation for API validity constraints. By integrating runtime feedback to automatically generate labeled training data, the method learns accurate usage patterns without relying on manual annotations. Implemented within the ACETest framework, the approach achieves over 91% classification accuracy across 183 APIs and significantly improves test pass rates—from 29% to 61%—demonstrating enhanced precision and scalability in testing deep learning libraries.

This work proposes a training-free method for constructing neural networks that precisely simulate arbitrary Turing machines. Given a formal description of a Turing machine, the system automatically compiles it into a corresponding neural network whose forward pass exactly mirrors a single step of the machine’s execution. Grounded in first principles, the approach employs ReLU networks to implement Boolean logic and adders, leverages Cantor set encoding together with hard attention mechanisms to manage tape read–write operations, and demonstrates Turing completeness within both Transformer architectures—incorporating self-attention and cross-attention—and recurrent neural network frameworks. This study establishes a reproducible and verifiable theoretical and practical foundation for integrating neural and symbolic computation.

Industrial-scale recommendation and ranking models feature highly complex and continuously evolving architectures, rendering traditional optimization approaches—based on manual intervention or module-level rules—difficult to scale. This work proposes the first extensible and customizable operator-level automatic transformation framework integrated into PyTorch 2.x. By leveraging FX intermediate representation, the PT2 compiler, predefined pattern matching, and a greedy search algorithm, the framework achieves general-purpose model optimizations while strictly preserving computational semantics. Evaluated on real-world industrial recommendation models, the approach delivers up to 63% inference speedup, a 6% reduction in peak memory usage, and over 400 seconds of compilation time savings. The implementation has been open-sourced as part of PyTorch 2.x.

This work addresses the inefficiency of forward and backward propagation in shallow neural networks on GPUs, which stems from memory access bottlenecks. To mitigate this issue, the authors propose a three-tier CUDA-based optimization strategy that integrates tiled shared-memory padding, pre-transposition of weight matrices, and fusion of matrix multiplication (MatMul) with ReLU activation operators. By enhancing memory access patterns, minimizing global memory round trips, and increasing computational density, the approach achieves a 1.41× speedup on an NVIDIA Tesla T4 GPU—reducing execution time from 21.0 seconds to 14.8 seconds. The optimized implementation significantly outperforms both serial CPU and OpenMP-parallelized baselines, demonstrating the effectiveness of the proposed techniques in accelerating training for shallow neural networks.