Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework

2025-02-26

AI Summary

Existing LLM-based automatic evaluation methods rely on predefined, generic criteria, limiting generalization to unseen instructions and exhibiting insufficient robustness in quantitative and structural constraint assessment. To address these limitations, we propose ARJudge—a novel “Analysis–Refinement” framework. The Analyzer module fine-tunes an open-source LLM to adaptively generate multidimensional evaluation criteria; the Refiner module performs zero-shot joint discrimination of numerical accuracy, format compliance, and other structural constraints by integrating semantic analysis with code execution verification. ARJudge is trained on a composite corpus covering three task categories: criterion generation, textual analysis, and code analysis. Extensive experiments demonstrate that ARJudge significantly outperforms state-of-the-art fine-tuned evaluators across multiple benchmarks, particularly improving generalization to unseen instructions and enhancing discrimination accuracy for numerical precision and syntactic/format constraints.

Technology Category

• Machine Learning: Evaluation and Analysis
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) are being used more and more extensively for automated evaluation in various scenarios. Previous studies have attempted to fine-tune open-source LLMs to replicate the evaluation explanations and judgments of powerful proprietary models, such as GPT-4. However, these methods are largely limited to text-based analyses under predefined general criteria, resulting in reduced adaptability for unseen instructions and demonstrating instability in evaluating adherence to quantitative and structural constraints. To address these limitations, we propose a novel evaluation framework, ARJudge, that adaptively formulates evaluation criteria and synthesizes both text-based and code-driven analyses to evaluate LLM responses. ARJudge consists of two components: a fine-tuned Analyzer that generates multi-faceted evaluation analyses and a tuning-free Refiner that combines and refines all analyses to make the final judgment. We construct a Composite Analysis Corpus that integrates tasks for evaluation criteria generation alongside text-based and code-driven analysis generation to train the Analyzer. Our results demonstrate that ARJudge outperforms existing fine-tuned evaluators in effectiveness and robustness. Furthermore, it demonstrates the importance of multi-faceted evaluation and code-driven analyses in enhancing evaluation capabilities.

Problem

• Align multi-faceted LLM evaluation criteria.
• Enhance adaptability to unseen instructions.
• Combine text and code-driven analysis effectively.

Innovation

• Adaptive evaluation criteria formulation
• Text-based and code-driven synthesis
• Fine-tuned Analyzer and tuning-free Refiner