Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction

2026-07-14

AI Summary

This study addresses the high cost of manually constructing reproducibility scoring rubrics, which hinders the scalability of benchmarks like PaperBench. It presents the first systematic evaluation of the reliability of large language models (LLMs) in automatically generating such rubrics, reformulated as checklists, using two backbone LLMs across four generation settings. Through internal and external meta-evaluations based on semantic similarity and alignment with human benchmarks, the authors find that enhanced generation strategies significantly improve downstream evaluation alignment, with the best configuration approaching human-level performance. Nevertheless, limitations persist in fine-grained coverage, scoring bias, and domain adaptability, and improvements in intrinsic semantic quality remain modest.

Abstract

Rubric-based evaluation is a promising approach for assessing open-ended outputs from LLM-based research agents, particularly in paper reproduction, where direct paper-to-repository comparison is prone to hallucination. However, constructing paper-specific rubrics requires substantial expert effort, limiting the scalability of benchmarks such as PaperBench. In this work, we present, to our knowledge, the first systematic meta-evaluation of LLM-generated rubrics for paper reproduction. We reformulate rubrics into a checklist-style format and evaluate four generation settings across two backbone models. We meta-evaluate generated rubrics intrinsically by semantic similarity and extrinsically by score alignment with ground-truth rubrics. Our results show that the augmented settings substantially improves downstream evaluation alignment, with the strongest setting approaching the human baseline, while intrinsic gains are more modest. Further analyses reveal that LLM-generated rubrics are often overly fine-grained, biased toward high scores, and less adaptive to paper domains, highlighting both the affordances and limitations.

Problem

• rubric generation
• LLM evaluation
• paper reproduction
• benchmark scalability
• open-ended output assessment

Innovation

• LLM-generated rubrics
• meta-evaluation
• checklist-style rubrics
• paper reproduction
• score alignment