Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners

2026-06-23

AI Summary

This work addresses the limited systematic relational reasoning capabilities of current neural models when confronted with complex instances outside their training distribution, as well as the absence of automated benchmarks for evaluating generalization. The authors propose an end-to-end framework that leverages large language models (LLMs) in conjunction with FunSearch-based evolutionary search and autonomous agents to automatically generate challenging relational reasoning tasks and discover effective hard-example sampling functions. This approach achieves, for the first time, fully automated construction of relational reasoning benchmarks within Datalog rule worlds, substantially enhancing the out-of-distribution and perturbation robustness of Edge Transformers. Furthermore, the generated tasks demonstrate transferability to novel reasoning environments introduced by LLMs, advancing neural relational reasoning toward an autonomous research paradigm.

Abstract

Reasoning about relational structures remains a significant challenge for neural models, particularly when they must systematically apply learned knowledge to problem instances that are harder than those seen in training. Progress is hampered by the difficulty of evaluating such generalization, since a priori, it is rarely clear what makes an instance hard. We study how this issue can be addressed by using large language models (LLMs) to automate benchmark generation, learning to produce increasingly challenging instances in an end-to-end manner. Concretely, given a world parametrized by Datalog rules, and an Edge Transformer as the reasoning evaluator, we use LLM-driven evolutionary search (based on FunSearch) and autonomous agentic search to discover sampling functions that yield hard problem instances. We also show that the Edge Transformer can be improved using this data such that it generalizes well to further data perturbations. Finally, we show that the same machinery can be applied to novel worlds proposed by LLMs, opening the door to autonomous research on neural relational reasoning.

Problem

• relational reasoning
• neural generalization
• benchmarking
• hard instance generation
• systematic generalization

Innovation

• automated benchmarking
• neural relational reasoning
• LLM-driven evolutionary search
• Edge Transformer
• systematic generalization