AFFORDANCE20Q: Evaluating Affordance Reasoning from Physical Properties

2026-06-12

AI Summary

This work addresses a critical limitation in existing affordance reasoning benchmarks, which inadvertently reveal object identities and thereby encourage models to rely on memorization rather than physical properties for inference. To overcome this, the authors introduce Affordance20Q, a novel benchmark that evaluates a model’s ability to infer 59 distinct affordances based solely on object attributes such as shape and material, while concealing object identity through a 20-question yes/no guessing game. The benchmark comprises 454 objects and 1,009 games. Furthermore, the study proposes Knowledge-base Anchored Rule Induction (KARI), a new method that leverages external knowledge to enhance rule-based reasoning. Experimental results demonstrate that current large language models underperform humans by approximately 20 points on this task, yet KARI improves the performance of open-source models by up to 15.2 points.

Abstract

Affordance reasoning, the inference of an object's action possibilities from its physical properties (e.g., shape and material), is fundamental to human physical understanding and increasingly critical for Large Language Models (LLMs). However, existing affordance benchmarks largely expose explicit object identities in the evaluation setup, allowing models to rely on memorized object-affordance mappings rather than reasoning over physical properties. To address this gap, we introduce Affordance20Q, a novel affordance reasoning benchmark formulated as a 20-Questions game without exposing the object's identity. In each game, the model identifies a hidden object's affordance from a candidate set by asking yes/no questions about its physical properties. Affordance20Q comprises 1,009 games over 454 objects and 59 affordances, all manually filtered, refined, and annotated. We conduct comprehensive experiments with 15 state-of-the-art LLMs and find a substantial gap (~20 points) compared to human performance. A KL-based information-gain (IG) analysis further shows that models fail to ask discriminating questions as the game progresses. To close the gap, we develop KB-Anchored Rule Induction (KARI), a pipeline based on LLMs that generates affordance rules grounded in evidence from knowledge bases (KBs). KARI improves open-source LLMs by up to 15.2 points, while the limited coverage of KBs hinders further gains. We release all our code and data at https://github.com/1171-jpg/Affordance20Q.git

Problem

• affordance reasoning
• physical properties
• object identity
• Large Language Models
• benchmark

Innovation

• Affordance Reasoning
• Physical Properties
• 20-Questions Game
• Knowledge Base Anchoring
• Rule Induction