Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results

2026-06-12

AI Summary

This work addresses the challenge of incomparable, non-reusable, and fragmented AI evaluation results stemming from heterogeneous formats, disparate sources, and inconsistent frameworks. To overcome this, the authors propose the first community-governed unified standard that defines JSON Schemas for both metadata and instance-level evaluation results, alongside a source-agnostic architecture and automated conversion tools supporting 31 diverse evaluation formats. Leveraging this standard, they have constructed a large-scale, standardized database encompassing 22,235 models and 2,273 benchmarks, hosted on Hugging Face with support for crowdsourced contributions. This infrastructure significantly enhances the comparability and reusability of evaluation results while fostering efficient cross-community collaboration.

Abstract

AI evaluations are widely used for testing and understanding progress. However, the diverse evaluators bring with them inconsistencies that challenge analysis and comparison. First, results are saved in incompatible formats, scattered across leaderboards, papers, blog posts, evaluation harness logs, and custom repositories. Second, results are created by different evaluation frameworks, which produce divergent scores for nominally identical evaluations and record metadata inconsistently, hindering comparison, cross-community evaluation science, cost reduction, and reuse. We introduce Every Eval Ever, the first shared schema and community-crowdsourced repository for AI evaluation results. The schema standardizes how evaluations are represented in a unified, single JSON document. It is source-agnostic by design, ingesting results from evaluation harnesses and papers alike, and optionally stores per-instance outputs for fine-grained analysis. We contribute: (i) a community-governed metadata schema with a companion instance-level schema, the first standardization effort of its kind; (ii) automatic converters from popular formats, evaluation harnesses, and leaderboards to the unified schema; and (iii) a crowdsourced community database hosted on Hugging Face, currently spanning to date 22,235 models, 2,273 unique benchmarks, and 31 evaluation formats.

Problem

• AI evaluation
• result standardization
• evaluation framework inconsistency
• cross-community comparison
• metadata interoperability

Innovation

• evaluation schema
• standardization
• community repository
• AI benchmarking
• metadata interoperability