An Empirical Study on Challenges for LLM Application Developers

2024-08-06

ACM Transactions on Software Engineering and Methodology

AI Summary

LLM application developers face significant, under-investigated challenges across prompt engineering, API integration, and plugin development. Method: We conducted a manual thematic analysis of 2,364 annotated samples drawn from 29,057 OpenAI Forum Q&A threads, constructing the first comprehensive, empirically grounded taxonomy of LLM development challenges—spanning design, implementation, and deployment phases. Contribution/Results: The taxonomy comprises seven high-level categories and 32 fine-grained subcategories, revealing critical pain points including strong platform dependency, lack of debugging tooling, and context management difficulties. Based on these findings, we propose 15 actionable, evidence-based recommendations targeting both developers and platform providers. This work establishes the first large-scale empirical foundation and methodological framework for LLM engineering practice.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In recent years, large language models (LLMs) have seen rapid advancements, significantly impacting various fields such as computer vision, natural language processing, and software engineering. These LLMs, exemplified by OpenAI’s ChatGPT, have revolutionized the way we approach language understanding and generation tasks. However, in contrast to traditional software development practices, LLM development introduces new challenges for AI developers in design, implementation, and deployment. These challenges span different areas (such as prompts, APIs, and plugins), requiring developers to navigate unique methodologies and considerations specific to LLM application development. Despite the profound influence of LLMs, to the best of our knowledge, these challenges have not been thoroughly investigated in previous empirical studies. To fill this gap, we present the first comprehensive study on understanding the challenges faced by LLM developers. Specifically, we crawl and analyze 29,057 relevant questions from a popular OpenAI developer forum. We first examine their popularity and difficulty. After manually analyzing 2,364 sampled questions, we construct a taxonomy of challenges faced by LLM developers. Based on this taxonomy, we summarize a set of findings and actionable implications for LLM-related stakeholders, including developers and providers (especially the OpenAI organization).

Problem

• Large Language Models
• Developer Challenges
• Effective Solutions

Innovation

• Large Language Models
• Developer Challenges
• Practical Recommendations