HPSS: Heuristic Prompting Strategy Search for LLM Evaluators

2025-02-18

AI Summary

This work addresses the insufficient joint optimization of multiple factors in LLM-based text evaluation prompt design. We propose a genetic algorithm–driven heuristic search framework that, for the first time, jointly models eight critical prompt factors—including evaluation criteria, output format, and example construction—as a unified, searchable space, enabling end-to-end automated prompt strategy optimization. The method is both efficient and scalable, requiring neither manual trial-and-error nor gradient computation. Evaluated on four mainstream text evaluation tasks, our approach significantly outperforms handcrafted prompts and existing automated baselines, achieving an average 12.6% improvement in Spearman correlation with human judgments. Key contributions include: (1) establishing a multi-factor coupled optimization paradigm for evaluation prompts; (2) introducing the first genetic search framework specifically designed for LLM evaluation prompt optimization; and (3) empirically validating its strong cross-task generalization capability.

Technology Category

• Search and Optimization: Evaluation and Analysis
• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Since the adoption of large language models (LLMs) for text evaluation has become increasingly prevalent in the field of natural language processing (NLP), a series of existing works attempt to optimize the prompts for LLM evaluators to improve their alignment with human judgment. However, their efforts are limited to optimizing individual factors of evaluation prompts, such as evaluation criteria or output formats, neglecting the combinatorial impact of multiple factors, which leads to insufficient optimization of the evaluation pipeline. Nevertheless, identifying well-behaved prompting strategies for adjusting multiple factors requires extensive enumeration. To this end, we comprehensively integrate 8 key factors for evaluation prompts and propose a novel automatic prompting strategy optimization method called Heuristic Prompting Strategy Search (HPSS). Inspired by the genetic algorithm, HPSS conducts an iterative search to find well-behaved prompting strategies for LLM evaluators. A heuristic function is employed to guide the search process, enhancing the performance of our algorithm. Extensive experiments across four evaluation tasks demonstrate the effectiveness of HPSS, consistently outperforming both human-designed evaluation prompts and existing automatic prompt optimization methods.

Problem

• Optimize prompts for LLM evaluators
• Enhance alignment with human judgment
• Automate combinatorial factor optimization

Innovation

• Heuristic Prompting Strategy Search
• Genetic algorithm-inspired optimization
• Integration of 8 key factors