A Systematic Survey of Automatic Prompt Optimization Techniques

2025-02-24

AI Summary

The rapid advancement of large language models has raised the barrier to effective prompting and hindered reproducibility in prompt engineering. To address this, this paper conducts a systematic study of Automated Prompt Optimization (APO). We formally define APO and propose the first unified five-dimensional framework—comprising objective, input, optimizer, feedback, and evaluation—derived from comprehensive literature analysis and cross-method feature abstraction. Our classification system is exhaustive and dimensionally explicit. We identify common APO challenges, including evaluation bias, poor generalization, and high computational overhead, and construct a structured knowledge graph to unify terminology and evaluation standards. The work establishes a scalable theoretical foundation and practical guidelines for APO, significantly enhancing the systematicity, rigor, and comparability of research in this emerging field.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Constraint Optimization

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Since the advent of large language models (LLMs), prompt engineering has been a crucial step for eliciting desired responses for various Natural Language Processing (NLP) tasks. However, prompt engineering remains an impediment for end users due to rapid advances in models, tasks, and associated best practices. To mitigate this, Automatic Prompt Optimization (APO) techniques have recently emerged that use various automated techniques to help improve the performance of LLMs on various tasks. In this paper, we present a comprehensive survey summarizing the current progress and remaining challenges in this field. We provide a formal definition of APO, a 5-part unifying framework, and then proceed to rigorously categorize all relevant works based on their salient features therein. We hope to spur further research guided by our framework.

Problem

• Automatic Prompt Optimization techniques
• Improve Large Language Models' performance
• Survey current progress and challenges

Innovation

• Automated prompt optimization techniques
• Comprehensive survey of APO
• Unifying framework for categorization