Agentified Assessment of Logical Reasoning Agents

2026-03-03

AI Summary

This work addresses the lack of reproducibility, auditability, and robustness to execution failures in current evaluations of logical reasoning agents. To overcome these limitations, the authors propose an “agentified evaluation” paradigm that models the evaluation process itself as agent behavior, introducing a standardized framework capable of task dispatching, execution budget control, output parsing, and structured failure logging. Built upon a unified agent interface, the framework integrates Z3Py-based automated formalization, SMT solving, and failure classification to enable automated, structured, and auditable assessment. Evaluated on a cleaned FOLIO validation set, the automated formalization agent achieves an accuracy of 86.70%, substantially outperforming a chain-of-thought baseline at 73.89%.

Technology Category

• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

We present a framework for evaluating and benchmarking logical reasoning agents when assessment itself must be reproducible, auditable, and robust to execution failures. Building on agentified assessment, we use an assessor agent to issue tasks, enforce execution budgets, parse outputs, and record structured failure types, while the agent under test only needs to expose a standardized agent-to-agent interface. As a case study, we benchmark an auto-formalization agent for first-order logic (FOL) reasoning on a solver-verified and repaired split of FOLIO. The agent translates natural language premises and conclusions into executable Z3Py programs and employs satisfiability modulo theories (SMT) solving to determine logical entailment. On the cleaned FOLIO validation set, the auto-formalization agent achieves 86.70% accuracy under the assessor protocol, outperforming a chain-of-thought baseline (73.89%).

Problem

• logical reasoning agents
• agentified assessment
• benchmarking
• reproducibility
• robust evaluation

Innovation

• agentified assessment
• logical reasoning agents
• auto-formalization
• SMT solving
• structured failure analysis