caffe

This work addresses the bottleneck in large-scale deep learning training, which often stems from system implementation rather than model architecture. To this end, the authors develop a lightweight, general-purpose training framework natively implemented in C++ and CUDA from first principles, integrating tensor computation, reverse-mode automatic differentiation, an efficient caching allocator, multi-modal distributed execution, and an MLIR-based compiler. This design achieves fine-grained hardware control while preserving modeling simplicity. On an 8-GPU RTX 6000 Ada system, the framework trains a 124-million-parameter GPT-2 model at 407K tokens/s—surpassing PyTorch’s 395K tokens/s—while reducing memory consumption by 22% and achieving a lower validation loss. This represents the first fully native, end-to-end tunable high-performance training system built solely on standard C++ and core CUDA primitives.

This work investigates the robustness of privacy-preserving unlearnability in Convolutionally Unlearnable Data (CUDA) under common post-processing operations—specifically, image sharpening and frequency-domain filtering. We systematically evaluate whether such transformations preserve CUDA’s claimed unlearnability. Contrary to prior assumptions, we find that simple enhancements—e.g., sharpening combined with low-frequency boosting—effectively restore discriminative features in CUDA samples, substantially improving downstream model training performance. On CIFAR-10, CIFAR-100, and ImageNet-100, models trained on recovered data consistently outperform those trained via adversarial training baselines. This study provides the first empirical evidence that CUDA-style unlearnability is highly fragile to standard image augmentations, challenging the foundational security assumptions of existing unlearnable data paradigms. Our findings offer critical insights for data poisoning defense, privacy-preserving machine learning, and robust data governance, establishing a new empirical foundation for evaluating unlearnability under realistic preprocessing threats.

This work proposes an efficient and scalable tabular foundation model to address the limitations of existing tabular prediction approaches in generalization, efficiency, and scalability. Built upon the Transformer architecture, the model introduces several key innovations: a high-diversity synthetic data generation strategy, a scalable Softmax attention mechanism, and, for the first time in tabular modeling, the adoption of the Muon optimizer. Remarkably, the model achieves strong out-of-the-box generalization without fine-tuning, surpassing the current state-of-the-art method, RealTabPFN-2.5, on both TabArena and TALENT benchmarks. It supports inference on datasets with up to one million samples while consuming less than 50 GB of GPU memory and delivers significantly improved inference speed.

This study addresses the GPU computational efficiency bottleneck in deep and machine learning. Methodologically, it proposes a task-aware GPU parallel architecture adaptation framework that systematically integrates CUDA stream-based concurrency, dynamic parallelism, and heterogeneous hardware (FPGA/TPU/ASIC) co-selection—implemented via deep integration into PyTorch, TensorFlow, and XGBoost. Its core contribution lies in establishing a transferable GPU optimization methodology, transcending model- or library-specific tuning. Experimental evaluation demonstrates 3–8× speedup across representative training and inference workloads. Furthermore, the authors open-source a modular, well-documented GPU optimization practice guide, substantially lowering the barrier to entry for AI practitioners seeking parallelization optimizations.

Existing certified training algorithms suffer from inconsistent evaluation protocols and suboptimal hyperparameter tuning, leading to incomparable performance claims and unreliable SOTA conclusions. Method: We introduce CTBENCH—the first unified benchmark for certified training—enabling fair, cross-algorithm evaluation of mainstream methods (e.g., IBP, CROWN-IBP, DeepPoly) under a standardized training pipeline, consistent ℓ∞/ℓ2 certification framework, and systematic hyperparameter optimization (grid search + Bayesian optimization). Contribution/Results: Our evaluation reveals that most recently proposed algorithms are substantially overestimated in prior work; after baseline enhancement, their relative improvements drop by over 40% on average. Crucially, all methods achieve significantly higher certified accuracy on CTBENCH than reported in their original papers. This work establishes a reproducible, extensible standard for evaluating certified training, redefining both the robustness training baseline and the SOTA landscape.

Training ultra-large language models (100B+ parameters) in full precision on a single GPU is constrained by limited device memory and CPU–GPU bandwidth. This work proposes a memory-centric training architecture that keeps model parameters and optimizer states in host memory, treating the GPU as a transient compute unit. By streaming layer parameters with a double-buffered pipeline and employing a stateless automatic differentiation mechanism, the approach eliminates persistent device state and overcomes the bandwidth bottleneck. The method successfully trains a 120B-parameter model on a single H200 GPU and achieves 1.84× higher throughput than DeepSpeed ZeRO-3 with CPU offloading for a 14B-parameter model. It also enables efficient training of a 7B-parameter model with a 512k-token context length on a single GH200 system.

This work addresses the challenge that large language models often violate strict execution constraints when generating CUDA kernels, leading to functional errors. The authors propose CuSeT, a low-cost instruction fine-tuning approach that is the first to uncover the dual manifestation of CUDA sensitivity at both token-level and region-level granularities. Building on this insight, CuSeT introduces a synergistic optimization strategy that integrates adaptive token-level masking with region-aware sample reweighting within a supervised fine-tuning framework, effectively balancing the exploitation of high-confidence tokens and the preservation of low-confidence yet critical regions. Experiments demonstrate that CuSeT consistently improves the functional correctness of generated kernels across multiple model families and scales, outperforming both standard and enhanced supervised fine-tuning baselines while achieving state-of-the-art performance at lower inference cost.

This work addresses the challenge of defect detection in deep learning libraries such as TensorFlow and PyTorch, where complex APIs often lead to subtle bugs and existing testing approaches suffer from high false-positive rates due to imprecise specifications. To overcome this limitation, the authors propose a machine learning classifier that leverages tensor shape abstraction as a precise input representation for API validity constraints. By integrating runtime feedback to automatically generate labeled training data, the method learns accurate usage patterns without relying on manual annotations. Implemented within the ACETest framework, the approach achieves over 91% classification accuracy across 183 APIs and significantly improves test pass rates—from 29% to 61%—demonstrating enhanced precision and scalability in testing deep learning libraries.

This work addresses the lack of transparent, scalable, and deeply PyTorch-integrated open-source tools for post-training large language models, which hinders research iteration and deployment efficiency. We propose a native PyTorch-based, modular post-training framework centered on the principle of “hackability,” offering composable model builders, training recipes, and a distributed training stack that support diverse fine-tuning strategies and hardware configurations. While maintaining high performance and memory efficiency, the framework significantly enhances code transparency and research flexibility. Empirical evaluations demonstrate that it matches or even surpasses mainstream tools such as Axolotl and Unsloth across multiple post-training scenarios, thereby facilitating efficient and reproducible scientific exploration.

This study investigates the generalization capability and overfitting behavior of neural networks on the CIFAR-10 image classification task. By constructing and comparing a fully connected network with a convolutional architecture comprising six convolutional layers and three max-pooling layers, the work implements a complete pipeline encompassing data preprocessing (normalization and one-hot encoding), training (using the Adam optimizer with mini-batches), and validation. After ten training epochs, the model achieves a validation accuracy of 74.77% and clearly exhibits the hallmark overfitting pattern: training loss continues to decrease while validation loss begins to rise. The findings underscore the distinction between representation learning and mere memorization, offering a reproducible benchmark framework that can inform the development of regularization techniques, data augmentation strategies, and educational experimentation.