LiveBench: A Challenging, Contamination-Limited LLM Benchmark

2024-06-27

AI Summary

Current LLM evaluation faces two critical challenges: test-set contamination and biases inherent in human- or LLM-based judgments. To address these, we propose the first contamination-resistant, human- and LLM-judgment-free dynamic benchmark. It dynamically curates high-difficulty real-world problems monthly from competitive programming contests, arXiv, news outlets, and other authoritative sources, covering six core domains—mathematics, code generation, logical reasoning, and more. Our framework introduces a novel triple-protection mechanism: (1) real-time source updating, (2) fully automated scoring grounded in objective ground truth, and (3) continuous task difficulty evolution. Leveraging dynamic data acquisition, structured annotation, automated evaluation pipelines, and cross-domain task composition, it integrates and strengthens existing benchmarks including Big-Bench Hard, AMPS, and IFEval. Evaluation across 100+ mainstream models (0.5B–405B parameters) reveals top-performing models still achieve <70% accuracy. All questions, reference solutions, and answers are open-sourced to enable community co-development and longitudinal capability tracking.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Test set contamination, wherein test data from a benchmark ends up in a newer model's training set, is a well-documented obstacle for fair LLM evaluation and can quickly render benchmarks obsolete. To mitigate this, many recent benchmarks crowdsource new prompts and evaluations from human or LLM judges; however, these can introduce significant biases, and break down when scoring hard questions. In this work, we introduce a new benchmark for LLMs designed to be resistant to both test set contamination and the pitfalls of LLM judging and human crowdsourcing. We release LiveBench, the first benchmark that (1) contains frequently-updated questions from recent information sources, (2) scores answers automatically according to objective ground-truth values, and (3) contains a wide variety of challenging tasks, spanning math, coding, reasoning, language, instruction following, and data analysis. To achieve this, LiveBench contains questions that are based on recently-released math competitions, arXiv papers, news articles, and datasets, and it contains harder, contamination-limited versions of tasks from previous benchmarks such as Big-Bench Hard, AMPS, and IFEval. We evaluate many prominent closed-source models, as well as dozens of open-source models ranging from 0.5B to 405B in size. LiveBench is difficult, with top models achieving below 70% accuracy. We release all questions, code, and model answers. Questions are added and updated on a monthly basis, and we release new tasks and harder versions of tasks over time so that LiveBench can distinguish between the capabilities of LLMs as they improve in the future. We welcome community engagement and collaboration for expanding the benchmark tasks and models.

Problem

• Addressing test set contamination in LLM evaluation
• Reducing biases from human and LLM judges
• Creating a dynamic, challenging benchmark for LLMs

Innovation

• Frequently-updated questions from recent sources
• Automated scoring with objective ground-truth
• Diverse challenging tasks contamination-resistant