Evaluating Chain-of-Thought Reasoning through Reusability and Verifiability

2026-02-19

AI Summary

This work addresses the limitation of current Chain-of-Thought (CoT) evaluation, which overly relies on task accuracy and fails to capture the intrinsic quality of the reasoning process itself. To overcome this, the authors propose reusability and verifiability as novel evaluation dimensions and introduce a Thinker-Executor framework that decouples reasoning generation from execution. Leveraging a multi-agent information retrieval architecture, they systematically evaluate four classes of Thinker models and ten Executor variants across five benchmarks. Experimental results reveal that the proposed metrics exhibit no significant correlation with conventional accuracy measures, and specialized reasoning models do not consistently outperform general-purpose large language models—such as Llama and Gemma—in CoT quality. These findings highlight critical blind spots in existing CoT evaluation paradigms.

Technology Category

• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In multi-agent IR pipelines for tasks such as search and ranking, LLM-based agents exchange intermediate reasoning in terms of Chain-of-Thought (CoT) with each other. Current CoT evaluation narrowly focuses on target task accuracy. However, this metric fails to assess the quality or utility of the reasoning process itself. To address this limitation, we introduce two novel measures: reusability and verifiability. We decouple CoT generation from execution using a Thinker-Executor framework. Reusability measures how easily an Executor can reuse the Thinker's CoT. Verifiability measures how frequently an Executor can match the Thinker's answer using the CoT. We evaluated four Thinker models against a committee of ten Executor models across five benchmarks. Our results reveal that reusability and verifiability do not correlate with standard accuracy, exposing a blind spot in current accuracy-based leaderboards for reasoning capability. Surprisingly, we find that CoTs from specialized reasoning models are not consistently more reusable or verifiable than those from general-purpose LLMs like Llama and Gemma.

Problem

• Chain-of-Thought
• reasoning evaluation
• reusability
• verifiability
• multi-agent IR

Innovation

• Chain-of-Thought
• reusability
• verifiability
• Thinker-Executor framework
• multi-agent reasoning