Do You Get the Hint? Benchmarking LLMs on the Board Game Concept

2025-10-15

AI Summary

Large language models (LLMs) exhibit significant weaknesses in abductive and abstract reasoning—particularly in iteratively modeling opponents’ strategic intentions and revising initial hypotheses during multi-turn interactions. Method: We introduce Concept, the first natural-language, board-game-style word-guessing benchmark designed to evaluate such capabilities. Concept formalizes reasoning as a turn-based adversarial dialogue grounded in real-world linguistic distributions. Our evaluation framework incorporates human baseline studies, fine-grained multi-turn dialogue analysis, and cross-lingual assessment across English, Dutch, French, and Spanish. Contribution/Results: Humans achieve >90% success, whereas state-of-the-art LLMs score <40%, with further degradation in low-resource languages. Concept exposes fundamental limitations in LLMs’ capacity for iterative intention inference and hypothesis revision—capabilities critical for higher-order reasoning. It establishes a novel, linguistically grounded paradigm for evaluating multilingual abstract reasoning, moving beyond static, single-shot benchmarks toward dynamic, interactive, and cognitively plausible evaluation protocols.

Technology Category

• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Large language models (LLMs) have achieved striking successes on many benchmarks, yet recent studies continue to expose fundamental weaknesses. In particular, tasks that require abstract reasoning remain challenging, often because they use representations such as grids, symbols, or visual patterns that differ from the natural language data LLMs are trained on. In this paper, we introduce Concept, a simple word-guessing board game, as a benchmark for probing abductive reasoning in a representation that is much closer to LLM pre-training data: natural language. Our results show that this game, easily solved by humans (with a success rate of over 90%), is still very challenging for state-of-the-art LLMs (no model exceeds 40% success rate). Specifically, we observe that LLMs struggle with interpreting other players' strategic intents, and with correcting initial hypotheses given sequential information updates. In addition, we extend the evaluation across multiple languages, and find that the LLM performance drops further in lower-resource languages (Dutch, French, and Spanish) compared to English.

Problem

• Benchmarking LLMs on abstract reasoning tasks
• Evaluating LLM performance in strategic intent interpretation
• Assessing LLM adaptability across multiple languages

Innovation

• Used board game Concept as benchmark
• Tested abductive reasoning in natural language
• Evaluated LLMs across multiple languages