Instance-level Performance Prediction for Long-form Generation Tasks

2025-09-08

AI Summary

This paper addresses the challenge of instance-level performance prediction for long-text generation tasks. We propose a task-, model-, and metric-agnostic black-box method that predicts multi-dimensional, fine-grained quality scores—such as factuality and coherence—as well as their predictive intervals, solely from input text and model output, thereby quantifying prediction uncertainty. Our key contributions are: (1) the first benchmark for instance-level performance prediction targeting multi-dimensional quality assessment; (2) a few-shot generalizable framework requiring only 16 annotated instances to achieve robust cross-task (11 tasks), cross-model (multiple LLMs), and cross-metric prediction; and (3) a unified architecture jointly modeling continuous-score regression and uncertainty estimation. Experiments demonstrate significant improvements over baselines. We release both an off-the-shelf tool and the open-source benchmark, advancing the practicality and trustworthiness of generative model performance forecasting.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Generation
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We motivate and share a new benchmark for instance-level performance prediction of long-form generation tasks having multi-faceted, fine-grained quality metrics. Our task-, model- and metric-agnostic formulation predicts continuous evaluation metric scores given only black-box model inputs and outputs. Beyond predicting point estimates of metric scores, the benchmark also requires inferring prediction intervals to quantify uncertainty around point estimates. Evaluation spans 11 long-form datasets/tasks with multiple LLMs, baselines, and metrics per task. We show that scores can be effectively predicted across long-form generation tasks using as few as 16 training examples. Overall, we introduce a novel and useful task, a valuable benchmark to drive progress, and baselines ready for practical adoption today.

Problem

• Predicting continuous evaluation metric scores for long-form generation tasks
• Inferring prediction intervals to quantify uncertainty in metric estimates
• Developing task- and model-agnostic performance prediction using minimal training examples

Innovation

• Black-box performance prediction for generation tasks
• Uncertainty quantification via prediction intervals
• Few-shot learning with minimal training examples