A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation

2026-05-17

AI Summary

Existing abstract reasoning benchmarks are limited by manual annotation, insufficient scale, and an inability to disentangle model memorization from genuine reasoning capabilities. This work proposes A2RBench, the first automated framework for generating abstract reasoning benchmarks with formal verifiability. It leverages large language models (LLMs) to create tasks requiring authentic reasoning, scales task variants by reusing formally verified rules to expand the input space, and incorporates cycle consistency and formal uniqueness verification to mitigate LLM hallucinations. Experimental results show that state-of-the-art LLMs perform substantially worse than humans on this benchmark (39.8% vs. 68.5%), particularly struggling with high-dimensional tasks, while inputs of higher informational complexity surprisingly facilitate reasoning.

Abstract

Abstract reasoning ability reflects the intelligence and generalization capacity of LLMs to extract and apply abstract rules. However, accurately measuring this ability remains challenging: existing benchmarks either rely on expensive manual annotation, limiting their scale, or risk measuring memorization rather than genuine reasoning. To address this, we introduce an automated pipeline named A2RBench, encompassing generation, expansion, evaluation, and analysis. Specifically, in the generation stage, LLMs create diverse tasks demanding genuine reasoning; in the expansion stage, LLMs reuse validated rules and expand new input spaces to generate task variations, achieving scaling. However, such a process may cause hallucinations. To eliminate it, we further establish a theoretical framework and prove that programmatic verification--testing whether the inverse operation perfectly reverses the forward operation (cycle consistency)--guarantees a unique solution. Through extensive evaluations on mainstream LLMs, we find: (1) Current LLMs exhibit fundamental deficiencies in abstract reasoning, with top models significantly underperforming humans on a representative subset (39.8% vs. 68.5%). (2) Current LLMs fall far short of 2D and 1D in the complexity of generated 3D tasks, revealing their lack of understanding of high-dimensional tasks. (3) Counterintuitively, inputs with higher information complexity can simplify the reasoning process.

Problem

• abstract reasoning
• benchmark generation
• formal verification
• LLM evaluation
• reasoning generalization

Innovation

• formal verification
• abstract reasoning
• automated benchmark generation
• cycle consistency
• LLM evaluation