Evaluating Relational Reasoning in LLMs with REL

2026-04-13

AI Summary

This work addresses the limitation of existing evaluations for large language models in relational reasoning, which predominantly rely on structured inputs and fail to disentangle the intrinsic difficulty of higher-order relational binding. To overcome this, the authors propose REL, a generative benchmark grounded in relational complexity (RC), which systematically modulates RC across algebra, chemistry, and biology to isolate and quantify, for the first time, the impact of higher-order relational binding on model performance. Through a theoretical framework of relational complexity, carefully designed generative tasks, and analyses incorporating in-context learning and test-time compute augmentation, the study reveals that the performance of mainstream models declines monotonically with increasing RC. Crucially, this bottleneck persists despite additional reasoning steps or exemplars, exposing a fundamental limitation in their capacity for higher-order relational reasoning.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Statistical Relational/Logic Learning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Relational reasoning is the ability to infer relations that jointly bind multiple entities, attributes, or variables. This ability is central to scientific reasoning, but existing evaluations of relational reasoning in large language models often focus on structured inputs such as tables, graphs, or synthetic tasks, and do not isolate the difficulty introduced by higher-arity relational binding. We study this problem through the lens of Relational Complexity (RC), which we define as the minimum number of independent entities or operands that must be simultaneously bound to apply a relation. RC provides a principled way to vary reasoning difficulty while controlling for confounders such as input size, vocabulary, and representational choices. Building on RC, we introduce REL, a generative benchmark framework spanning algebra, chemistry, and biology that varies RC within each domain. Across frontier LLMs, performance degrades consistently and monotonically as RC increases, even when the total number of entities is held fixed. This failure mode persists with increased test-time compute and in-context learning, suggesting a limitation tied to the arity of the required relational binding rather than to insufficient inference steps or lack of exposure to examples. Our results identify a regime of higher-arity reasoning in which current models struggle, and motivate re-examining benchmarks through the lens of relational complexity.

Problem

• relational reasoning
• relational complexity
• higher-arity binding
• large language models
• reasoning evaluation

Innovation

• relational reasoning
• relational complexity
• higher-arity binding
• generative benchmark
• large language models