Prompting for Performance: Exploring LLMs for Configuring Software

2025-07-13

AI Summary

Software performance optimization faces an exponentially large configuration space, rendering traditional machine learning approaches computationally expensive and heavily reliant on domain expertise. Method: This paper presents the first systematic investigation of large language models (LLMs) for performance-aware configuration across representative systems—including compilers, video encoders, and SAT solvers—and introduces a novel prompt-engineering-based paradigm comprising three tasks: configuration option identification, performance ranking, and recommendation. Contribution/Results: Experiments demonstrate that LLMs achieve near-expert accuracy on simple-to-moderately complex configuration tasks, validating their feasibility as lightweight configuration assistants. However, they exhibit hallucination and superficial reasoning in deep inference scenarios. The study delineates the practical potential and inherent limitations of LLMs in software performance configuration, establishing a foundation for low-overhead, expert-free automated tuning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Algorithm Configuration
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Software systems usually provide numerous configuration options that can affect performance metrics such as execution time, memory usage, binary size, or bitrate. On the one hand, making informed decisions is challenging and requires domain expertise in options and their combinations. On the other hand, machine learning techniques can search vast configuration spaces, but with a high computational cost, since concrete executions of numerous configurations are required. In this exploratory study, we investigate whether large language models (LLMs) can assist in performance-oriented software configuration through prompts. We evaluate several LLMs on tasks including identifying relevant options, ranking configurations, and recommending performant configurations across various configurable systems, such as compilers, video encoders, and SAT solvers. Our preliminary results reveal both positive abilities and notable limitations: depending on the task and systems, LLMs can well align with expert knowledge, whereas hallucinations or superficial reasoning can emerge in other cases. These findings represent a first step toward systematic evaluations and the design of LLM-based solutions to assist with software configuration.

Problem

• LLMs assist in performance-oriented software configuration
• Evaluating LLMs for ranking and recommending configurations
• Exploring LLMs' abilities and limitations in software configuration

Innovation

• Using LLMs for software configuration optimization
• Evaluating LLMs on ranking and recommending configurations
• Exploring LLM alignment with expert knowledge