model training & foundation models

Large language models (LLMs) face significant challenges in full-parameter fine-tuning under constrained GPU memory and computational resources, hindering efficient adaptation to downstream tasks. To address this, this work systematically surveys parameter-efficient fine-tuning (PEFT) methodologies and proposes the first unified conceptual framework—comprehensively covering theoretical foundations, algorithmic taxonomies (e.g., LoRA, Adapter, Prompt/Prefix Tuning), cross-modal extensions, and emerging trends. Distinct from fragmented surveys, our framework explicitly articulates theoretical interconnections and practical applicability boundaries across methods, unifying representative paradigms from both NLP and multimodal learning. We further release an open-source, structured knowledge graph encoding these insights. The resulting framework substantially lowers the barrier to lightweight LLM adaptation, offering researchers and practitioners a reusable, transferable technical guide. By bridging theoretical analysis with engineering pragmatism, this work accelerates the transition of PEFT from methodological exploration to scalable, production-ready deployment.

This study systematically evaluates the applicability of the foundation model (FM) paradigm across three scientific domains—genomics, satellite imagery, and time-series analysis—to assess whether FMs can supplant traditional supervised learning. Method: We construct a cross-modal benchmark framework employing lightweight architectures (e.g., Wide ResNet, U-Net), automated hyperparameter optimization, and standardized training protocols to rigorously compare domain-specific FMs against strong supervised baselines. Contribution/Results: Across all tasks, carefully tuned supervised models match or exceed state-of-the-art domain-specific FMs; large-scale pretraining yields no consistent empirical gains. This work provides the first multi-modal scientific validation that the FM paradigm remains immature for these domains. We open-source two automated evaluation workflows and underscore the necessity—and benchmarking value—of strong supervised baselines in scientific AI assessment.

The theoretical mechanisms underlying parameter-efficient fine-tuning (PEFT) methods for large pre-trained models remain poorly understood, and the performance disparities among existing approaches lack principled explanations. Method: This paper establishes, for the first time, a unified theoretical framework grounded in matrix decomposition, revealing that diverse PEFT methods fundamentally perform optimization under low-rank constraints. Leveraging this insight, we propose two novel PEFT methods and a general-purpose enhancement framework—designed with theoretical rigor and architectural generality—through SVD- and LoRA-style modeling analysis, modular design, and multi-task empirical validation. Contribution/Results: Our approach significantly improves the performance of canonical PEFT methods—including LoRA and Adapter—across mainstream NLP benchmarks. This work provides the first principle-level, systematic explanation of PEFT and establishes an extensible technical pathway for future advancements.

How can large pre-trained models be efficiently deployed in federated learning while balancing communication efficiency and model performance? This paper introduces FedPEFT, the first systematic integration of parameter-efficient fine-tuning (PEFT) into federated learning. In FedPEFT, clients update only a small set of trainable modules—e.g., LoRA or Adapters—while the server performs lightweight aggregation of these sparse updates. The framework natively accommodates practical constraints including Non-IID data distributions, client dropouts, and differential privacy requirements. Extensive experiments across multiple federated benchmarks demonstrate that FedPEFT reduces total communication overhead by up to 95% compared to standard baselines, while matching or surpassing the accuracy of FedAvg. These results significantly enhance the practical feasibility of deploying large language models in resource-constrained edge environments.

To address the high communication overhead and limited compute-communication overlap in tensor model parallelism (TMP) for large language models, this paper proposes Oases, a dependency-aware, fine-grained TMP auto-optimization framework. Our method introduces: (1) operator-level fine-grained training scheduling; (2) communication-computation overlap–aware modeling and planning; (3) the first automated TMP strategy search algorithm supporting dependency constraints; and (4) end-to-end runtime optimization for TMP. Evaluated across multiple models (e.g., LLaMA, BERT) and hardware platforms (A100/H100), Oases achieves 1.01–1.48× speedup over state-of-the-art approaches and up to 1.9× improvement over Megatron-LM. These results demonstrate significant gains in training efficiency for large-scale foundation models.

To address the high communication overhead and weak privacy guarantees in federated fine-tuning of large foundation models over distributed data, this paper proposes a novel one-shot aggregation paradigm. Theoretically and empirically, we establish—for the first time—that large models can achieve performance comparable to multi-round federated fine-tuning without iterative parameter aggregation. Our method integrates loss surface analysis with cross-domain distributed optimization, enabling asynchronous local updates and built-in differential privacy enhancement. Evaluated on text generation and text-to-image tasks, our one-shot fine-tuning attains over 98% of the performance of conventional multi-round approaches while reducing communication costs by more than 90%. This significantly improves training efficiency, privacy preservation, and system flexibility—particularly under resource-constrained and heterogeneous federated settings.

This work addresses the challenge of national supercomputing centers struggling to efficiently support the full lifecycle of foundation models—including pretraining, fine-tuning, and inference—by proposing a hybrid cloud-native platform that integrates diskless GPU-accelerated HPE Cray EX nodes with virtualized general-purpose infrastructure. Leveraging Kubernetes for unified orchestration, the platform bridges traditional HPC batch processing and AI-serving workflows, enabling, for the first time in a national supercomputing environment, an end-to-end “AI factory” architecture for foundation models. This approach effectively closes the paradigm gap between high-performance computing and cloud-native AI services, substantially enhancing user productivity and offering a reusable implementation blueprint for integrating end-to-end AI applications into supercomputing centers.

To address the dual heterogeneity—diverse model architectures and downstream tasks—in Hybrid Heterogeneous Federated Fine-Tuning (HHFFT), which causes dimension mismatch and multi-task knowledge interference, this paper proposes the first systematic solution: (1) a sparsified triple matrix decomposition to achieve aggregable low-rank representations of heterogeneous LoRA parameters; (2) a relation-guided layer alignment mechanism to mitigate architectural discrepancies across clients; and (3) an alternating task knowledge disentanglement framework to separate shared and task-specific knowledge. Theoretically, we prove the algorithm converges at rate O(1/√T). Empirically, our method achieves up to 15.4% higher accuracy than state-of-the-art approaches across multiple benchmarks, significantly improving cross-device knowledge sharing efficiency and personalization performance.

Current governance paradigms typically assume that the safety properties of foundation models are preserved after fine-tuning; however, this assumption lacks systematic validation in high-stakes domains such as healthcare and law. This study presents the first comprehensive, multidimensional safety evaluation of 100 fine-tuned models—including widely deployed domain-specific models and controlled experimental variants—across both general and domain-specific benchmarks. The findings reveal that fine-tuning frequently induces significant and heterogeneous shifts in safety: improvements along certain metrics often coincide with severe degradation in others. These results challenge the prevailing practice of relying solely on base model safety assessments and underscore the critical need for independent, thorough safety re-evaluation of fine-tuned models before deployment in high-risk applications.

This study systematically investigates the effectiveness and applicability of fine-tuning in Tabular Foundation Models (TFMs), aiming to answer when and how fine-tuning can improve performance, calibration, and fairness. Through comprehensive evaluation across benchmarks including TALENT, OpenML-CC18, and TabZilla, the work compares zero-shot inference, meta-learning, full supervised fine-tuning (SFT), and parameter-efficient fine-tuning (PEFT). The findings reveal that zero-shot TFMs already exhibit strong performance, while SFT often degrades accuracy or calibration quality. Meta-learning and PEFT yield only marginal gains under specific data conditions. This work is the first to demonstrate that fine-tuning is not universally beneficial and provides practical, data-characteristic-driven guidelines for its application, clearly delineating the boundaries of its advantages.

Existing large-scale model training systems struggle to flexibly compose diverse parallelization strategies, often relying on manual expert tuning and lacking generality. This work proposes a programmable distributed training system that enables users to declaratively specify composite parallelism strategies—such as data, pipeline, and expert parallelism—through model annotations and scheduling directives. These specifications are compiled via a unified intermediate representation (IR) into device-level execution plans, fully decoupling strategy definition from runtime execution over a global compute-communication DAG. The system is the first to support automatic compilation of user-defined composite strategies, matching the performance of established approaches like ZeRO while significantly improving both performance and memory efficiency in complex scenarios such as DeepSeek-V3’s DualPipe.