RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques

2025-01-24

AI Summary

Existing evaluation frameworks for LLMs’ critical reasoning capabilities lack rigorous methodologies to distinguish genuine differences in critical analysis and self-correction efficacy between advanced reasoning models and standard ones. Method: We introduce the first correction-oriented, closed-loop benchmark for critical reasoning evaluation, featuring three novel mechanisms—self-critique, cross-model critique, and iterative critique—quantifying critique effectiveness through multi-round correction quality. Built upon eight challenging reasoning tasks, the benchmark integrates critique-correction pipeline validation, cross-model comparative analysis, and iterative feedback assessment. Contribution/Results: Experiments reveal that conventional LLMs systematically underperform o1-mini across all critique tasks; notably, in self-critique and iterative critique settings, their corrected outputs degrade relative to the original responses. This work establishes a reproducible, scalable paradigm for evaluating LLMs’ critical thinking abilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Critiques are important for enhancing the performance of Large Language Models (LLMs), enabling both self-improvement and constructive feedback for others by identifying flaws and suggesting improvements. However, evaluating the critique capabilities of LLMs presents a significant challenge due to the open-ended nature of the task. In this work, we introduce a new benchmark designed to assess the critique capabilities of LLMs. Unlike existing benchmarks, which typically function in an open-loop fashion, our approach employs a closed-loop methodology that evaluates the quality of corrections generated from critiques. Moreover, the benchmark incorporates features such as self-critique, cross-critique, and iterative critique, which are crucial for distinguishing the abilities of advanced reasoning models from more classical ones. We implement this benchmark using eight challenging reasoning tasks. We have several interesting findings. First, despite demonstrating comparable performance in direct chain-of-thought generation, classical LLMs significantly lag behind the advanced reasoning-based model o1-mini across all critique scenarios. Second, in self-critique and iterative critique settings, classical LLMs may even underperform relative to their baseline capabilities. We hope that this benchmark will serve as a valuable resource to guide future advancements. The code and data are available at url{https://github.com/tangzhy/RealCritic}.

Problem

• Large Language Models
• Critical Thinking
• Performance Evaluation

Innovation

• RealCritic
• Critical Reasoning Assessment
• Advanced Language Models