Auto-PRE: An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation

2024-10-16

AI Summary

To address bottlenecks in LLM evaluation—including high human annotation costs, rigid task formatting, reliance on reference answers, and systemic biases—this paper proposes AutoReview, an automated peer-review framework. Methodologically, AutoReview introduces the first LLM auto-selection mechanism grounded in three dimensions: consistency, relevance, and confidence—spanning instruction understanding, content alignment, and response discrimination. It integrates the LLM-as-judge paradigm, multi-dimensional capability quantification, structured qualification exams, and task-adaptive evaluator selection. Empirically, AutoReview achieves state-of-the-art performance on three diverse tasks—summarization, non-factual QA, and dialogue generation—while substantially reducing evaluation cost. Moreover, it demonstrates strong scalability and cross-task generalization capability, enabling robust, reference-free, and format-agnostic LLM assessment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The rapid development of large language models (LLMs) has highlighted the need for efficient and reliable methods to evaluate their performance. Traditional evaluation methods often face challenges like high costs, limited task formats, dependence on human references, and systematic biases. To address these limitations, we propose Auto-PRE, an automatic LLM evaluation framework inspired by the peer review process. Unlike previous approaches that rely on human annotations, Auto-PRE automatically selects evaluator LLMs based on three core traits: consistency, pertinence, and self-confidence, which correspond to the instruction, content, and response stages, respectively, and collectively cover the entire evaluation process. Experiments on three representative tasks, including summarization, non-factoid QA, and dialogue generation, demonstrate that Auto-PRE achieves state-of-the-art performance while significantly reducing evaluation costs. Furthermore, the structured and scalable design of our automatic qualification exam framework provides valuable insights into automating the evaluation of LLMs-as-judges, paving the way for more advanced LLM-based evaluation frameworks.

Problem

• Automates LLM evaluation to reduce human annotation costs
• Addresses systematic biases in traditional language generation assessment
• Selects optimal evaluator models through multi-stage qualification traits

Innovation

• Automatically selects evaluator LLMs for peer review
• Uses consistency, pertinence, and self-confidence traits
• Achieves state-of-the-art performance with reduced costs