model deployment

This study addresses the lack of a systematic review on machine learning (ML) method selection and application in monolith-to-microservices migration. We conducted a rigorous systematic literature review (SLR) of 81 studies published between 2015 and 2024, adhering to the PRISMA guidelines. Our primary contribution is the first four-dimensional classification framework—encompassing migration phases, input data types, ML techniques, and evaluation practices—which reveals that stages such as monitoring and service identification have reached initial maturity, whereas critical phases like microservice packaging remain severely underexplored. We identify three core challenges: data scarcity, poor scalability of proposed approaches, and the absence of standardized benchmarks. The framework establishes a unified analytical paradigm for the field and provides concrete directions for future research and practical implementation of ML-driven migration strategies.

This study empirically characterizes, for the first time, the full lifecycle practices of pre-trained models (PTMs) in open-source software (OSS), focusing on integration, evolution, testing, and maintenance challenges. We conduct large-scale mining of GitHub repositories, coupled with cross-platform PTM dependency tracing (Hugging Face, PyTorch Hub), historical commit and issue log analysis, and model metadata parsing. Our analysis systematically identifies recurring risks—including dependency staleness, inadequate documentation, and insufficient test coverage. We propose a novel software engineering analysis framework specifically designed for model dependencies, addressing critical gaps in PTM operationalization and sustainability research. As concrete outcomes, we deliver a reusable PTM maintenance practice guide and a prototype detection tool. These contributions provide both theoretical foundations and practical support for enhancing the maintainability and engineering rigor of AI models within software systems.

To address the fragmented model optimization workflows and poor cross-stack compatibility in the PyTorch ecosystem, this paper introduces the first end-to-end native AI model optimization framework. It establishes a unified abstraction for low-precision tensors—supporting FP8, quantization-aware training (QAT), post-training quantization (PTQ), and 2:4 sparsity—thereby closing the loop across pretraining, fine-tuning, quantization, and deployment. A novel tensor subclass design enables seamless backend interoperability and deep integration with major toolchains, including TorchTitan/Tune, vLLM, and Hugging Face Transformers. The framework significantly improves engineering efficiency and compression consistency. It has been successfully applied to quantized-sparse variants of Llama 3.2 (1B/3B) and LlamaGuard3-8B, demonstrating practical efficacy. The open-source implementation is publicly available on GitHub, enabling efficient, full-pipeline deployment.

Misuse of machine learning (ML) cloud services is widespread in practice, degrading system quality and maintainability, yet lacks a unified definition or taxonomy. Method: We conduct a multi-source empirical study—integrating a systematic literature review, cloud provider documentation analysis, code mining from 377 GitHub projects, and surveys plus thematic coding with 50 practitioners—to systematically characterize such misuse. Contribution/Results: We introduce the first comprehensive taxonomy of 20 ML cloud service misuses, 16 of which are newly identified. We propose the “multi-voice empirical paradigm,” unifying academic, industrial, open-source, and survey evidence. Our findings reveal that misuses frequently occur in real-world deployments, primarily due to misconceptions about service capabilities and inadequate documentation. This work establishes the first scalable, empirically grounded foundation for ML engineering education, best-practice guidelines, and automated detection tools.

This work addresses customs clearance delays in global trade caused by ambiguous product descriptions and frequent updates to Harmonized System (HS) codes. To tackle this challenge, the authors propose a serverless MLOps framework that leverages event-driven pipelines and managed services to enable end-to-end, model-agnostic machine learning lifecycle management. The architecture supports automatic scaling, reproducible training, auditable deployment, and automated A/B testing, ensuring secure and seamless model transitions. By integrating custom text embeddings with models such as Text-CNN, the system achieves 98% accuracy on real-world HS code prediction tasks, meeting stringent service-level agreement (SLA) requirements. This approach significantly reduces long-term operational costs and establishes an efficient, cost-effective, and reproducible deployment paradigm for industrial-scale machine learning systems.

This work addresses the lack of transparent, scalable, and deeply PyTorch-integrated open-source tools for post-training large language models, which hinders research iteration and deployment efficiency. We propose a native PyTorch-based, modular post-training framework centered on the principle of “hackability,” offering composable model builders, training recipes, and a distributed training stack that support diverse fine-tuning strategies and hardware configurations. While maintaining high performance and memory efficiency, the framework significantly enhances code transparency and research flexibility. Empirical evaluations demonstrate that it matches or even surpasses mainstream tools such as Axolotl and Unsloth across multiple post-training scenarios, thereby facilitating efficient and reproducible scientific exploration.

This work addresses the challenges of SLO violations and resource inefficiency in machine learning model serving caused by inadequate capacity planning. To this end, the authors propose an adaptive, feedback-driven load testing framework that formalizes the ML serving load testing process for the first time. The framework incorporates real-traffic-based workload calibration and a warm-up mechanism, combined with adaptive search, performance signal feedback control, convergence detection, and GPU monitoring to efficiently estimate the maximum sustainable throughput under SLO constraints. Evaluation across 14 industrial cases demonstrates that the approach reduces capacity estimation error from approximately 30% to 2–6%, with the warm-up mechanism improving accuracy by 22.2%. This significantly mitigates deployment incidents and enhances GPU resource utilization efficiency.

Online serving of any-to-any multimodal models faces significant challenges due to extreme heterogeneity in request types, computation paths, and resource requirements. Method: This paper proposes the first automated deployment planning framework supporting generic computational graph modeling, enabling model-component-level dynamic decoupling, heterogeneous hardware-aware scheduling, and distributed runtime co-optimization. Contribution/Results: The framework systematically addresses the service challenge of arbitrary combinations of multimodal inputs (text, image, video, audio) to multimodal outputs. Evaluated against state-of-the-art baselines, it achieves up to a 3.81× throughput improvement and up to a 5.79× reduction in P99 latency, significantly enhancing service efficiency and scalability while maintaining end-to-end correctness and quality.

This study addresses the widespread practice among model-sharing platforms of permitting the loading of remote custom code, which introduces severe remote code execution (RCE) vulnerabilities. It presents the first systematic quantification of this risk across major platforms, combining static analysis tools (Bandit, CodeQL, Semgrep), YARA-based malicious pattern detection, API and documentation audits, and qualitative analysis of over 600 community discussions. The findings reveal pervasive issues including insecure default configurations, inconsistent policy enforcement, and insufficient developer awareness. To contextualize these vulnerabilities, the work establishes a CWE-based classification framework that elucidates the inherent tension between security and usability in model-sharing ecosystems. Based on this analysis, the paper offers concrete recommendations for strengthening the security of model-sharing infrastructure.

This work addresses the challenges of efficient inference in Any-to-Any multimodal models, which arise from the diverse combinations of input and output modalities, heterogeneous computational paths, and varying scalability requirements across components. To tackle these issues, the authors propose Cornserve, a distributed serving system tailored for such models. Cornserve employs a flexible task abstraction to express computation graphs, enabling component-level decoupling and independent scaling. It introduces a record-replay execution model combined with a direct tensor transfer mechanism from producers to consumers, optimizing both computation scheduling and data dependency-aware communication. Built on Kubernetes, Cornserve supports arbitrary modality combinations and demonstrates up to a 3.81× improvement in throughput and a 5.79× reduction in tail latency.

This work addresses the practical challenges of deploying machine learning models in real-world settings, where heterogeneous data protocols, non-standard formats, and infrastructure constraints often necessitate redundant construction of integration pipelines. To overcome these issues, we propose SMOCS—a containerized, streaming ML system built on Apache Kafka—that decouples infrastructure from application logic through layered abstraction and employs a three-threaded agent architecture to separate data ingestion, online training, and real-time inference. The framework enables configuration-driven, no-code deployment, offering platform independence, fault isolation, and horizontal scalability, thereby significantly lowering the barrier to entry for domain experts. SMOCS has been open-sourced on the Jefferson Lab GitHub repository and demonstrates both continuous online learning capability and strong engineering practicality.