CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion

2026-07-06

AI Summary

This work addresses the challenge of evaluating generative AI models under limited annotation budgets, where conventional approaches suffer from low statistical efficiency. The authors formulate model evaluation as a matrix completion problem and propose a novel framework that integrates low-rank matrix approximation, control variates, and prediction-driven inference to achieve unbiased estimation with substantially improved statistical efficiency. By leveraging historical evaluation data and the inherent dependencies among models on shared tasks, the method reconstructs a sparse scoring matrix and constructs reliable confidence intervals. Extensive experiments across diverse datasets, models, and sparsity regimes demonstrate that, under identical annotation budgets, the proposed approach significantly narrows confidence intervals and reduces the mean squared error of point estimates compared to existing baselines.

Abstract

Evaluating generative AI models is a routine, but resource-intensive, process that is conducted over and over again during the course of model development. In this work, we propose Collaborative Evaluation (CollabEval), a simple, effective, and principled method for exploiting dependencies between historical runs of different models on the same tasks to improve statistical efficiency. Specifically, our approach treats model evaluation as a matrix completion problem over an $M \times N$ matrix of evaluation scores, where $M$ is the total number of models and $N$ is the total number of evaluation prompts. We assume that a subset of these $M$ models are targeted for evaluation. For these target models only a small fraction, $p$, of prompts has been annotated with evaluation scores. Leveraging recent results in prediction-powered inference, we build a low-rank approximation of the score matrix, and use the reconstructed values as control variates in a manner that guarantees unbiased estimates of the true evaluation metric mean, in addition to statistically valid confidence intervals. Empirically, across a wide range of datasets, models, and sparsity levels $p$, we find that CollabEval substantially reduces the mean confidence interval size, and the mean squared error of the point estimate, compared to baseline methods at the same annotation budget.

Problem

• model evaluation
• statistical efficiency
• generative AI
• annotation budget
• confidence intervals

Innovation

• matrix completion
• collaborative evaluation
• statistical efficiency
• control variates
• prediction-powered inference