A Sequential Optimal Learning Approach to Automated Prompt Engineering in Large Language Models

2025-01-07

AI Summary

To address the high human cost and limited evaluation budget in large language model (LLM) prompt engineering, this paper proposes an automated prompt optimization framework based on sequential optimal learning. Methodologically, it introduces the forward Knowledge Gradient (KG) — for the first time — into prompt search, integrating feature-based prompt encoding with Bayesian regression to model prompt-performance correlations, thereby enabling scalable, constraint-aware optimization over large prompt spaces. The KG acquisition policy is efficiently solved via mixed-integer second-order cone programming (MISOCP). Empirically, on instruction induction tasks, the method achieves significantly better performance than multiple baselines using fewer evaluations, demonstrating KG’s superiority and practicality under tight evaluation budgets. Key contributions include: (i) the first application of forward KG to prompt optimization; (ii) a principled Bayesian modeling framework for prompt performance with expressive feature representations; and (iii) an efficient MISOCP-based solver for KG-driven prompt selection.

Technology Category

• Search and Optimization: Learning to Search
• Natural Language Processing: Prompt Engineering / Prompting
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Designing effective prompts is essential to guiding large language models (LLMs) toward desired responses. Automated prompt engineering aims to reduce reliance on manual effort by streamlining the design, refinement, and optimization of natural language prompts. This paper proposes an optimal learning framework for automated prompt engineering, designed to sequentially identify effective prompt features while efficiently allocating a limited evaluation budget. We introduce a feature-based method to express prompts, which significantly broadens the search space. Bayesian regression is employed to utilize correlations among similar prompts, accelerating the learning process. To efficiently explore the large space of prompt features for a high quality prompt, we adopt the forward-looking Knowledge-Gradient (KG) policy for sequential optimal learning. The KG policy is computed efficiently by solving mixed-integer second-order cone optimization problems, making it scalable and capable of accommodating prompts characterized only through constraints. We demonstrate that our method significantly outperforms a set of benchmark strategies assessed on instruction induction tasks. The results highlight the advantages of using the KG policy for prompt learning given a limited evaluation budget. Our framework provides a solution to deploying automated prompt engineering in a wider range applications where prompt evaluation is costly.

Problem

• Automated Prompt Design
• Large Language Models
• Cost-Efficiency

Innovation

• Bayesian Regression
• Knowledge Gradient Strategy
• Automated Prompt Engineering