How are MLOps Frameworks Used in Open Source Projects? An Empirical Characterization

2026-01-26

AI Summary

This study addresses the unclear usage patterns and functional demands surrounding current MLOps frameworks in open-source projects, which hinder their effective evolution. For the first time, it systematically links real-world framework adoption with user enhancement requests by analyzing GitHub dependencies, API invocations, and issue reports across eight prominent MLOps frameworks, employing qualitative coding and thematic mapping. The findings reveal that developers prefer customized integrations over out-of-the-box solutions, and that these frameworks are seldom directly embedded in GitHub Workflows, instead being primarily applied to core machine learning phases and infrastructure governance. Users most frequently request enhancements to core functionality, greater API exposure, and improved CI/CD integration, while increasingly adopting multiple frameworks in tandem.

Technology Category

• Machine Learning: Auto ML and Hyperparameter Tuning
• Search and Optimization: Algorithm Configuration
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations
• Responsible Web: Consent frameworks and practices on the web

Abstract

Machine Learning (ML) Operations (MLOps) frameworks have been conceived to support developers and AI engineers in managing the lifecycle of their ML models. While such frameworks provide a wide range of features, developers may leverage only a subset of them, while missing some highly desired features. This paper investigates the practical use and desired feature enhancements of eight popular open-source MLOps frameworks. Specifically, we analyze their usage by dependent projects on GitHub, examining how they invoke the frameworks'APIs and commands. Then, we qualitatively analyze feature requests and enhancements mined from the frameworks'issue trackers, relating these desired improvements to the previously identified usage features. Results indicate that MLOps frameworks are rarely used out-of-the-box and are infrequently integrated into GitHub Workflows, but rather, developers use their APIs to implement custom functionality in their projects. Used features concern core ML phases and whole infrastructure governance, sometimes leveraging multiple frameworks with complementary features. The mapping with feature requests highlights that users mainly ask for enhancements to core features of the frameworks, but also better API exposure and CI/CD integration.

Problem

• MLOps
• open-source projects
• framework usage
• feature requests
• empirical study

Innovation

• MLOps
• empirical study
• open-source frameworks
• API usage
• feature requests