EQUATOR: A Deterministic Framework for Evaluating LLM Reasoning with Open-Ended Questions. # v1.0.0-beta

2024-12-31

AI Summary

Existing evaluation methods for open-ended reasoning tasks suffer from fluency bias, hindering accurate assessment of factual accuracy and deep logical reasoning—thus remaining heavily reliant on manual annotation. This paper introduces the first deterministic LLM reasoning evaluation framework for open-ended questions: it constructs a gold-standard answer set via vector-database retrieval augmented with human annotation, and designs a rule-based automated scoring mechanism grounded in deterministic criteria. To enable end-to-end automation, the framework integrates a lightweight, locally deployed LLaMA-3.2B model (via Ollama). Experiments demonstrate that our framework achieves high accuracy while significantly outperforming conventional multiple-choice benchmarks; it improves evaluation scalability and reduces dependence on human scoring by over 70%. Its core innovation lies in enabling, for the first time, reproducible, interpretable, and low-dependency joint evaluation of factual correctness and logical reasoning in open-domain question answering.

Technology Category

• Natural Language Processing: Question Answering
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Despite the remarkable coherence of Large Language Models (LLMs), existing evaluation methods often suffer from fluency bias and rely heavily on multiple-choice formats, making it difficult to assess factual accuracy and complex reasoning effectively. LLMs thus frequently generate factually inaccurate responses, especially in complex reasoning tasks, highlighting two prominent challenges: (1) the inadequacy of existing methods to evaluate reasoning and factual accuracy effectively, and (2) the reliance on human evaluators for nuanced judgment, as illustrated by Williams and Huckle (2024)[1], who found manual grading indispensable despite automated grading advancements. To address evaluation gaps in open-ended reasoning tasks, we introduce the EQUATOR Evaluator (Evaluation of Question Answering Thoroughness in Open-ended Reasoning). This framework combines deterministic scoring with a focus on factual accuracy and robust reasoning assessment. Using a vector database, EQUATOR pairs open-ended questions with human-evaluated answers, enabling more precise and scalable evaluations. In practice, EQUATOR significantly reduces reliance on human evaluators for scoring and improves scalability compared to Williams and Huckle's (2004)[1] methods. Our results demonstrate that this framework significantly outperforms traditional multiple-choice evaluations while maintaining high accuracy standards. Additionally, we introduce an automated evaluation process leveraging smaller, locally hosted LLMs. We used LLaMA 3.2B, running on the Ollama binaries to streamline our assessments. This work establishes a new paradigm for evaluating LLM performance, emphasizing factual accuracy and reasoning ability, and provides a robust methodological foundation for future research.

Problem

• Large Language Models
• Evaluation Methods
• Factuality and Logic Assessment

Innovation

• EQUATOR Estimator
• Factuality and Reasoning Evaluation
• Automated Assessment with Local Models