What Slows Down FMware Development? An Empirical Study of Developer Challenges and Resolution Times

2025-10-13

AI Summary

This study presents the first large-scale empirical investigation into the core challenges and time-intensive bottlenecks in FMware (foundation model–based software) development. Addressing key problems—including heterogeneous target environments, nonstandardized development workflows, and immature toolchains—across cloud platforms and local open-source ecosystems (e.g., GitHub, Hugging Face, GPTStore), we adopt an empirical software engineering approach integrating quantitative analysis and qualitative interviews. Our findings identify memory management, dependency conflicts, and tokenizer configuration as the most frequent technical pain points; code review, semantic similarity search, and prompt template design as the most time-consuming activities; and education, content creation, and business strategy as the top three application domains. This work delivers the first cross-platform empirical evidence to inform the design of FMware-specific toolchains and optimize development practices.

Technology Category

• Application Domains: Software Engineering
• Machine Learning: Hardware-aware ML
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Foundation Models (FMs), such as OpenAI's GPT, are fundamentally transforming the practice of software engineering by enabling the development of emph{FMware} -- applications and infrastructures built around these models. FMware systems now support tasks such as code generation, natural-language interaction, knowledge integration, and multi-modal content creation, underscoring their disruptive impact on current software engineering workflows. However, the design, implementation, and evolution of FMware present significant new challenges, particularly across cloud-based and on-premise platforms where goals, processes, and tools often diverge from those of traditional software development. To our knowledge, this is the first large-scale analysis of FMware development across both cloud-based platforms and open-source repositories. We empirically investigate the FMware ecosystem through three focus areas: (1) the most common application domains of FMware, (2) the key challenges developers encounter, and (3) the types of issues that demand the greatest effort to resolve. Our analysis draws on data from GitHub repositories and from leading FMware platforms, including HuggingFace, GPTStore, Ora, and Poe. Our findings reveal a strong focus on education, content creation, and business strategy, alongside persistent technical challenges in memory management, dependency handling, and tokenizer configuration. On GitHub, bug reports and core functionality issues are the most frequently reported problems, while code review, similarity search, and prompt template design are the most time-consuming to resolve. By uncovering developer practices and pain points, this study points to opportunities to improve FMware tools, workflows, and community support, and provides actionable insights to help guide the future of FMware development.

Problem

• Identifies key challenges in FMware development across platforms
• Analyzes developer pain points and issue resolution times
• Investigates technical bottlenecks like memory and dependency management

Innovation

• Analyzed GitHub repositories and FMware platforms
• Identified key challenges in memory and dependency management
• Studied time-consuming issues in code review and prompts