DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models

2026-06-16

AI Summary

This work addresses defeasible abductive reasoning—a task requiring models to revise default assumptions rationally when explaining anomalous observations without undermining unrelated expectations—and introduces DeFAb, the first formal, verifiable, large-scale benchmark for this purpose. DeFAb automatically generates over 372,648 reasoning instances with polynomial-time verifiability by integrating knowledge bases such as OpenCyc, YAGO, and Wikidata with semantic graphs including ConceptNet and UMLS, and embeds the Lean 4 proof kernel to ensure logical rigor. The benchmark includes DeFAb-Hard and CONJURE variants, enabling judge-free evaluation and preference optimization. Experiments reveal that rule-based solvers achieve 100% accuracy in under 50 microseconds, whereas state-of-the-art language models reach at most 65%, dropping to 23.5% under robust evaluation, with a pronounced 19.4-percentage-point gap observed on Level 3 tasks.

Abstract

A rule-based logic solver resolves every instance in our benchmark in under 50 microseconds with 100% accuracy; the best frontier language model reaches 65% at best and drops to 23.5% under rendering-robust evaluation (worst case over four surface renderings). We introduce DeFAb (Defeasible Abduction Benchmark), a dataset and generation pipeline that converts four decades of publicly funded knowledge bases into formally grounded instances for defeasible abduction: constructing hypotheses that explain anomalies by overriding defaults while preserving unrelated expectations. Because every hypothesis must pass polynomial-time checks for valid derivation, conservativity, and minimality, DeFAb makes logical rigor the instrument for measuring creativity and theoretical reasoning, scoring the disciplined construction of theory revisions rather than fluent but theory-destroying prose. The pipeline pairs taxonomic hierarchies (OpenCyc, YAGO, Wikidata) with behavioral property graphs (ConceptNet, UMLS) to produce 372,648+ instances across 33.75M materialized rules from 18 sources, in three levels with polynomial-time verifiable gold standards. Four frontier models do not reliably internalize defeasible reasoning: rendering-robust Level 2 accuracy is 7.8-23.5%; chain-of-thought variance (~36 pp) exceeds any inter-model gap; and a matched contamination control isolates a +19.4 pp Level 3 gap. We further release DeFAb-Hard (a 235-instance Level 3 difficulty variant; best model 53.3% vs 100% symbolic) and CONJURE (a kernel-verified transformative-creativity variant of 560 Lean 4/Mathlib instances whose gold answers are definitions the proof kernel did not previously contain, judge-free verifier; a pilot finds zero novel concepts). The same verifier doubles as an exact reward for preference optimization (DPO, RLVR/GRPO). Released under MIT at https://huggingface.co/datasets/PatrickAllenCooper/DeFAb.

Problem

• defeasible abduction
• foundation models
• verifiable benchmark
• theoretical reasoning
• creativity evaluation

Innovation

• defeasible abduction
• verifiable benchmark
• polynomial-time verification
• foundation models
• preference optimization