ELPO: Ensemble Learning Based Prompt Optimization for Large Language Models

2025-11-20

AI Summary

Current large language models (LLMs) heavily rely on manually crafted prompts, while existing automatic prompt optimization (APO) methods suffer from weak generalizability and poor performance on complex tasks. To address these limitations, this paper proposes ELPO—an Ensemble Learning-based Prompt Optimization framework. ELPO innovatively integrates multi-strategy collaborative search—including evolutionary algorithms and trial-and-error mechanisms—with a shared generation mechanism, and introduces a voting-based ensemble strategy to enhance the robustness and adaptability of prompt generation. Evaluated on multi-task benchmarks including ArSarcasm, ELPO consistently outperforms state-of-the-art APO methods, achieving a 7.6-point improvement in F1 score. This demonstrates substantial advances in accuracy, stability, and cross-task generalization capability.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The remarkable performance of Large Language Models (LLMs) highly relies on crafted prompts. However, manual prompt engineering is a laborious process, creating a core bottleneck for practical application of LLMs. This phenomenon has led to the emergence of a new research area known as Automatic Prompt Optimization (APO), which develops rapidly in recent years. Existing APO methods such as those based on evolutionary algorithms or trial-and-error approaches realize an efficient and accurate prompt optimization to some extent. However, those researches focus on a single model or algorithm for the generation strategy and optimization process, which limits their performance when handling complex tasks. To address this, we propose a novel framework called Ensemble Learning based Prompt Optimization (ELPO) to achieve more accurate and robust results. Motivated by the idea of ensemble learning, ELPO conducts voting mechanism and introduces shared generation strategies along with different search methods for searching superior prompts. Moreover, ELPO creatively presents more efficient algorithms for the prompt generation and search process. Experimental results demonstrate that ELPO outperforms state-of-the-art prompt optimization methods across different tasks, e.g., improving F1 score by 7.6 on ArSarcasm dataset.

Problem

• Optimizing prompts automatically for large language models
• Overcoming limitations of single-method prompt optimization approaches
• Enhancing accuracy and robustness in complex task performance

Innovation

• Ensemble learning framework for prompt optimization
• Voting mechanism with shared generation strategies
• Efficient algorithms for prompt generation and search