neural networks

This work addresses the challenge of understanding backpropagation in Transformer architectures (e.g., GPT) for beginners. We systematically derive analytical gradients for core components—including token embeddings, multi-head self-attention, LayerNorm, and LoRA—using an index-free vectorized differentiation approach that avoids cumbersome subscript manipulations and yields lightweight, closed-form gradient expressions. To our knowledge, this is the first unified, end-to-end analytical derivation of full-chain backpropagation in Transformers, with explicit gradient formulas for LoRA fine-tuning. We accompany the analysis with a minimal runnable GPT implementation and provide closed-form solutions for all parameter updates. The results significantly enhance theoretical understanding of Transformer training dynamics and improve debugging capabilities, thereby establishing a rigorous foundation for pedagogy and interpretability research.

To address the O(n²) computational bottleneck of attention backpropagation in Transformer training for long sequences, this work proposes a controllable sparse backpropagation mechanism: during backward pass, gradients are retained for at most c attention connections per token per head, achieved via unbiased probabilistic pruning. We theoretically prove that the induced gradient variance increase decays with sequence length. Experiments show that, at n ≈ 2000, pruning rates reach 99% (c = 20–30), increasing gradient variance by only ~1% while reducing backward computation by ~100×—making backward latency comparable to forward latency. This is the first method to reduce attention backpropagation complexity to O(nc), significantly improving training efficiency for long sequences without modifying the forward pass or model architecture.

High barriers to adopting pre-trained models and a lack of empirical guidance for strategy selection hinder practical deployment in few-shot image classification and object detection. Method: We systematically compare linear probing versus fine-tuning across ResNet, MobileNet, and EfficientNet, and propose an end-to-end TensorFlow framework integrating multi-scale feature-space visualization (PCA, t-SNE, UMAP) to unify analysis of representation evolution. Contribution/Results: Linear probing significantly outperforms fine-tuning under extreme data scarcity (≤100 samples per class) while accelerating training by 3–5×. The framework enables high-accuracy, rapid deployment (<1 hour for fine-tuning) on standard benchmarks (ImageNet-1K, CIFAR-100), balancing beginner-friendly usability with expert-level extensibility. It bridges the gap between theoretical representation analysis and real-world engineering practice.

To address the non-convex optimization challenge in neural network classification—specifically, susceptibility to poor local minima and flat regions—this paper proposes SAGRAD, a batch-training algorithm integrating Simulated Annealing (SA) with Møller’s Scaled Conjugate Gradient (SCG) method. Its core innovation lies in the first incorporation of SA into the SCG framework, enabling a dynamic restart and escape mechanism that synergistically balances global exploration and local acceleration. Implemented in Fortran 77, SAGRAD incorporates efficient Hessian-vector multiplication, optimized gradient computation, and an adaptive SA weight initialization strategy. Empirical evaluation across multiple classification benchmarks demonstrates significantly improved convergence robustness and generalization performance, while markedly reducing the probability of converging to suboptimal local minima. These results validate SAGRAD’s effectiveness and practicality for non-convex optimization in neural network training.

This work investigates how two-layer neural networks adaptively learn the structure of target functions under high-dimensional Gaussian data via a small number of large-batch gradient descent steps, focusing on the interplay between batch size and iteration count, directional heterogeneity in learning difficulty, and required sample complexity. Method: Leveraging concentration inequalities, projection-based conditional analysis, and the Gaussian equivalence principle, we develop a neuron-direction-specific theoretical framework; we introduce the “staircase property” and “transition exponent” to rigorously quantify directional learning difficulty and sharply separate the performance regimes of feature learning and lazy training. Results: We prove that only $O(d)$ samples suffice to efficiently learn multiple target directions across iterations, achieving approximation and generalization errors substantially better than initialization. The learning process exhibits a distinctive staircase-like accuracy improvement—characterized by abrupt performance gains at critical iteration thresholds—demonstrating nontrivial adaptation to underlying function structure.

This work addresses the limited parallelizability of conventional Transformers as depth increases, which hinders training efficiency for large-scale models. The authors introduce, for the first time, a multilayer parallel-in-time algorithm into Transformer training by modeling the network as a neural ordinary differential equation (neural ODE), enabling cross-layer parallelism in both forward and backward passes. To ensure stable convergence while maximizing computational efficiency, they further propose an error-monitoring mechanism with adaptive switching between serial and parallel execution modes. Experiments on BERT, GPT-2, Vision Transformers (ViT), and machine translation architectures demonstrate that the method significantly enhances parallel scalability and training speed for deep models without compromising pretraining or fine-tuning accuracy.

This work proposes a biologically inspired, unsupervised personalization method that eliminates the need for backpropagation and labeled data, addressing the inefficiency of conventional deep neural networks in resource-constrained settings. Drawing inspiration from neural pruning mechanisms in the brain, the approach introduces a fine-grained, plasticity-driven pruning strategy into mainstream architectures such as ResNet-50 for the first time. Evaluated on benchmarks including ImageNet, the method achieves approximately 70% sparsity while boosting accuracy to around 90%, substantially reducing computational overhead. This demonstrates a notable departure from the typical trade-off between model efficiency and performance, as the technique simultaneously lowers resource consumption and enhances predictive accuracy.

Existing hardware-software co-design tools struggle to accurately model memory consumption and backward-pass complexity in neural network training. This work proposes the first extension of the experimentally validated inference modeling framework, Stream, to the training domain, introducing a comprehensive framework for modeling and optimizing training on heterogeneous dataflow accelerators. The framework supports training workflow modeling, exploration of layer fusion configurations, and optimization of activation checkpointing strategies. Integrated with a genetic algorithm for hardware architecture search, it is validated on ResNet-18 and a small-scale GPT-2 model, effectively uncovering critical trade-offs between performance and memory in training-specific hardware design and identifying superior architectures and training strategies.

This work proposes a training-free method for constructing neural networks that precisely simulate arbitrary Turing machines. Given a formal description of a Turing machine, the system automatically compiles it into a corresponding neural network whose forward pass exactly mirrors a single step of the machine’s execution. Grounded in first principles, the approach employs ReLU networks to implement Boolean logic and adders, leverages Cantor set encoding together with hard attention mechanisms to manage tape read–write operations, and demonstrates Turing completeness within both Transformer architectures—incorporating self-attention and cross-attention—and recurrent neural network frameworks. This study establishes a reproducible and verifiable theoretical and practical foundation for integrating neural and symbolic computation.

This work addresses the challenges of vanishing and exploding gradients in backpropagation by proposing a gradient-free training method for deep neural networks. Built upon an extremely simple Monte Carlo strategy, the approach randomly perturbs network parameters and retains updates that reduce the loss, enabling effective optimization on a single GPU. The study demonstrates, for the first time, that such a minimalist gradient-free algorithm can directly train networks exceeding 20 layers without relying on batch normalization or residual connections. Moreover, it is compatible with purely pruned architectures, discrete weights, and non-standard activation functions such as Gaussian activations. Experiments validate the method’s efficacy and generality by successfully training ultra-deep networks, wide networks with 16,384 neurons, and a minimal Transformer on MNIST and Tiny Shakespeare benchmarks.