CAPO: Cost-Aware Prompt Optimization

2025-04-22

AI Summary

Large language models (LLMs) exhibit strong sensitivity to prompt design, yet existing automated prompt optimization methods incur high API invocation costs and excessive token consumption. To address this, we propose an efficient multi-objective prompt optimization framework that jointly optimizes instruction templates, in-context examples, and prompt length. Our approach is the first to integrate AutoML-style racing mechanisms with multi-objective evolutionary algorithms for prompt search. We introduce task-description-guided initialization to enhance robustness and enable zero-shot cold-start optimization—i.e., without requiring any in-context examples. Leveraging the LLM itself as an evaluation operator, we incorporate length regularization and racing-based early termination to accelerate convergence. Evaluated across 15 benchmark tasks, our method outperforms state-of-the-art approaches on 11 tasks, achieving up to a 21-percentage-point improvement. Under identical computational budgets, it reduces total evaluations significantly, yields shorter average prompt lengths, and demonstrates strong robustness to initial prompt quality.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) have revolutionized natural language processing by solving a wide range of tasks simply guided by a prompt. Yet their performance is highly sensitive to prompt formulation. While automated prompt optimization addresses this challenge by finding optimal prompts, current methods require a substantial number of LLM calls and input tokens, making prompt optimization expensive. We introduce CAPO (Cost-Aware Prompt Optimization), an algorithm that enhances prompt optimization efficiency by integrating AutoML techniques. CAPO is an evolutionary approach with LLMs as operators, incorporating racing to save evaluations and multi-objective optimization to balance performance with prompt length. It jointly optimizes instructions and few-shot examples while leveraging task descriptions for improved robustness. Our extensive experiments across diverse datasets and LLMs demonstrate that CAPO outperforms state-of-the-art discrete prompt optimization methods in 11/15 cases with improvements up to 21%p. Our algorithm achieves better performances already with smaller budgets, saves evaluations through racing, and decreases average prompt length via a length penalty, making it both cost-efficient and cost-aware. Even without few-shot examples, CAPO outperforms its competitors and generally remains robust to initial prompts. CAPO represents an important step toward making prompt optimization more powerful and accessible by improving cost-efficiency.

Problem

• Optimizing prompts for LLMs efficiently
• Reducing cost of prompt optimization
• Balancing performance and prompt length

Innovation

• Integrates AutoML for efficient prompt optimization
• Uses evolutionary approach with LLM-based operators
• Balances performance and cost via multi-objective optimization