HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam

2026-02-15

AI Summary

This work addresses the severe noise in the original Humanity's Last Exam (HLE) benchmark, which introduces significant bias in model evaluation and distorts cross-model comparisons. To mitigate this issue, the authors propose a two-stage verification and repair pipeline that integrates a transparent validation protocol, a fine-grained error taxonomy, expert double-blind revision, and model-assisted auditing—ensuring precise corrections while preserving the original intent of each question. The resulting HLE-Verified benchmark comprises 641 verified questions, 1,170 revised and certified items, and 689 uncertain cases. Experimental results demonstrate that leading large language models achieve an average accuracy gain of 7–10 percentage points on this refined benchmark, with improvements reaching 30–40 percentage points on originally erroneous questions.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models on challenging, multi-domain questions. However, community-led analyses have raised concerns that HLE contains a non-trivial number of noisy items, which can bias evaluation results and distort cross-model comparisons. To address this challenge, we introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy. Our construction follows a two-stage validation-and-repair workflow resulting in a certified benchmark. In Stage I, each item undergoes binary validation of the problem and final answer through domain-expert review and model-based cross-checks, yielding 641 verified items. In Stage II, flawed but fixable items are revised under strict constraints preserving the original evaluation intent, through dual independent expert repairs, model-assisted auditing, and final adjudication, resulting in 1,170 revised-and-certified items. The remaining 689 items are released as a documented uncertain set with explicit uncertainty sources and expertise tags for future refinement. We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points on HLE-Verified. The improvement is particularly pronounced on items where the original problem statement and/or reference answer is erroneous, with gains of 30--40 percentage points. Our analyses further reveal a strong association between model confidence and the presence of errors in the problem statement or reference answer, supporting the effectiveness of our revisions. Overall, HLE-Verified improves HLE-style evaluations by reducing annotation noise and enabling more faithful measurement of model capabilities. Data is available at: https://github.com/SKYLENAGE-AI/HLE-Verified

Problem

• Humanity's Last Exam
• benchmark noise
• evaluation bias
• noisy items
• LLM evaluation

Innovation

• benchmark verification
• error taxonomy
• systematic revision
• model evaluation
• annotation noise reduction