The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans

2026-06-25

AI Summary

This study investigates whether large language models (LLMs) possess human-like flexible reasoning capabilities or merely rely on pattern matching from their training data. To this end, the authors introduce a novel “riddle puzzle” paradigm that systematically contrasts literal comprehension with the demands of creative inference, enabling a direct comparison of strategic adaptability between LLMs and humans. Drawing on methods from cognitive psychology and natural language processing, they conduct controlled experiments involving nine state-of-the-art LLMs and 100 human participants. Results reveal that while models achieve 84.9% accuracy on conventional riddles, their performance drops sharply to 50.7% on riddle puzzles; humans exhibit the opposite pattern (50.5% vs. 80.5%). Notably, model errors predominantly stem from overapplication of creative reasoning. This work provides the first empirical evidence of a fundamental limitation in LLMs’ reasoning flexibility.

Abstract

Humans flexibly adapt their reasoning strategies to the requirements of a given problem. Large language models (LLMs) have performed well on many cognitive tasks, however, it is unclear whether this accuracy is a result of pattern matching from training data or flexible reasoning. Here, we introduce a novel paradigm to test this question: the riddle riddle paradigm. Riddle riddles are word problems written to mimic popular riddles, but altered so their answers only require literal interpretations. Identifying correct answers requires looking past the structure of each question and flexibly apply different reasoning strategies based on the content. If LLMs respond to surface features, such as form, a riddle-like structure should cause models to use an inventive reasoning strategy even when a literal interpretation suffices. Alternatively, if LLMs reason based on content, they should flexibly switch strategies when appropriate. Across two experiments with nine state-of-the-art LLMs and 100 human participants, we show humans and LLMs fail on this paradigm in opposite directions. LLMs were far more accurate on genuine riddles than on riddle riddles (84.9% vs. 50.7%); whereas humans showed the reverse effect (50.5% vs. 80.5%). Error analysis shows that 90.8% of LLM errors on riddle riddles (the condition where they show diminished performance) were due to inappropriate use of inventive reasoning while only 57.6% of human errors on genuine riddles were due to overextending literal reasoning. Thus, while both groups make mistakes, reasoning mistakes are made more often by LLMs than by humans. Overall, LLMs' strong performance on genuine riddles may reflect memory retrieval rather than flexible strategy selection, and without stimuli designed to elicit this contrast, it becomes easy to conflate LLM-generated outputs that look like reasoning with genuine reasoning.

Problem

• flexible reasoning
• large language models
• riddle paradigm
• cognitive tasks
• reasoning strategies

Innovation

• flexible reasoning
• riddle riddle paradigm
• large language models
• cognitive evaluation
• reasoning strategy