"AI just keeps guessing": Using ARC Puzzles to Help Children Identify Reasoning Errors in Generative AI

2025-05-21

AI Summary

Children aged 6–11 often overtrust generative AI (genAI) due to its authoritative tone and structured outputs, even when reasoning is flawed. To address this, we designed AI Puzzlers—an interactive educational system that adapts Abstraction and Reasoning Corpus (ARC) tasks into age-appropriate, visual error-detection training grounded in Mayer and Moreno’s multimedia cognitive theory. The system employs dual-channel (visual–linguistic) presentation to reduce extraneous cognitive load and scaffold children’s autonomous identification of implicit reasoning errors and strategy development. Two rounds of participatory design (N=21) empirically uncovered children’s error-recognition behavioral patterns, common strategies, and credibility assessment mechanisms. Our contributions include: (1) the first framework for genAI reasoning error recognition tailored to children; (2) empirical validation of an ARC-based, cognitively adaptive design paradigm; and (3) actionable theoretical and practical foundations for AI literacy education.

Technology Category

• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Knowledge Representation and Reasoning: Argumentation
• Reasoning under Uncertainty: Causality

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The integration of generative Artificial Intelligence (genAI) into everyday life raises questions about the competencies required to critically engage with these technologies. Unlike visual errors in genAI, textual mistakes are often harder to detect and require specific domain knowledge. Furthermore, AI's authoritative tone and structured responses can create an illusion of correctness, leading to overtrust, especially among children. To address this, we developed AI Puzzlers, an interactive system based on the Abstraction and Reasoning Corpus (ARC), to help children identify and analyze errors in genAI. Drawing on Mayer&Moreno's Cognitive Theory of Multimedia Learning, AI Puzzlers uses visual and verbal elements to reduce cognitive overload and support error detection. Based on two participatory design sessions with 21 children (ages 6 - 11), our findings provide both design insights and an empirical understanding of how children identify errors in genAI reasoning, develop strategies for navigating these errors, and evaluate AI outputs.

Problem

• Helping children detect reasoning errors in generative AI outputs
• Addressing overtrust in AI due to authoritative tone and structured responses
• Reducing cognitive overload in children when identifying AI errors

Innovation

• Uses ARC Puzzles for error detection
• Applies Cognitive Theory of Multimedia Learning
• Interactive system for children's AI literacy