OLMES: A Standard for Language Model Evaluations

2024-06-12

arXiv.org

AI Summary

Current LLM evaluation lacks standardized practices—particularly in prompt design, in-context example selection, probability normalization, and task formalization—leading to poor reproducibility and unreliable cross-model comparisons. Method: We introduce OLMES, the first open-source, fully documented LLM evaluation standard. It systematically identifies and standardizes previously overlooked evaluation variables, enabling fair comparison between cloze-style and natural-language tasks. Grounded in empirical analysis, ablation studies, and comprehensive literature review, OLMES establishes technical specifications for prompt engineering, in-context learning configuration, logit calibration, and multi-granularity task adaptation. Contribution/Results: OLMES significantly improves the reliability, reproducibility, and comparability of LLM performance evaluations across model scales. By codifying best practices and resolving methodological inconsistencies, it advances LLM evaluation toward scientific rigor and community-wide standardization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Evaluation and Analysis

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Progress in AI is often demonstrated by new models claiming improved performance on tasks measuring model capabilities. Evaluating language models can be particularly challenging, as choices of how a model is evaluated on a task can lead to large changes in measured performance. There is no common standard setup, so different models are evaluated on the same tasks in different ways, leading to claims about which models perform best not being reproducible. We propose OLMES, a completely documented, practical, open standard for reproducible LLM evaluations. In developing this standard, we identify and review the varying factors in evaluation practices adopted by the community - such as details of prompt formatting, choice of in-context examples, probability normalizations, and task formulation. In particular, OLMES supports meaningful comparisons between smaller base models that require the unnatural"cloze"formulation of multiple-choice questions against larger models that can utilize the original formulation. OLMES includes well-considered, documented recommendations guided by results from existing literature as well as new experiments resolving open questions.

Problem

• Lack of standardized language model evaluation
• Irreproducible performance claims in AI
• Need for consistent LLM assessment methods

Innovation

• Standardized LLM evaluation framework
• Documented prompt formatting details
• Supports base and large model comparisons