Investigating Middle School Students Question-Asking and Answer-Evaluation Skills When Using ChatGPT for Science Investigation

2025-05-02

AI Summary

This study investigates the efficacy of 14–15-year-old lower-secondary students’ question formulation and critical evaluation of AI-generated answers during scientific inquiry using ChatGPT, addressing prevalent cognitive biases—including overreliance, vague questioning, and misjudgment of responses—and a potential metacognitive deficit. Method: A mixed-methods approach integrated interaction log analysis, open-ended problem-solving tasks, self-report questionnaires, and coded learning outcomes. Contribution/Results: Findings reveal a significant negative correlation between self-reported proficiency and actual question-answering (QA) performance. Crucially, this study provides the first empirical evidence that students’ metacognitive awareness strongly predicts both question quality and answer evaluation accuracy. Metacognition is thus identified as a core regulatory variable in AI literacy. These results establish a foundational theoretical framework and actionable pedagogical pathways for generative AI education targeting adolescents.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Search and Optimization: Metareasoning and Metaheuristics
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Generative AI (GenAI) tools such as ChatGPT allow users, including school students without prior AI expertise, to explore and address a wide range of tasks. Surveys show that most students aged eleven and older already use these tools for school-related activities. However, little is known about how they actually use GenAI and how it impacts their learning. This study addresses this gap by examining middle school students ability to ask effective questions and critically evaluate ChatGPT responses, two essential skills for active learning and productive interactions with GenAI. 63 students aged 14 to 15 were tasked with solving science investigation problems using ChatGPT. We analyzed their interactions with the model, as well as their resulting learning outcomes. Findings show that students often over-relied on ChatGPT in both the question-asking and answer-evaluation phases. Many struggled to use clear questions aligned with task goals and had difficulty judging the quality of responses or knowing when to seek clarification. As a result, their learning performance remained moderate: their explanations of the scientific concepts tended to be vague, incomplete, or inaccurate, even after unrestricted use of ChatGPT. This pattern held even in domains where students reported strong prior knowledge. Furthermore, students self-reported understanding and use of ChatGPT were negatively associated with their ability to select effective questions and evaluate responses, suggesting misconceptions about the tool and its limitations. In contrast, higher metacognitive skills were positively linked to better QA-related skills. These findings underscore the need for educational interventions that promote AI literacy and foster question-asking strategies to support effective learning with GenAI.

Problem

• Examining middle school students' question-asking skills with ChatGPT
• Assessing students' ability to evaluate ChatGPT responses critically
• Investigating impact of ChatGPT use on science learning outcomes

Innovation

• Analyzing middle school students ChatGPT interactions
• Evaluating question-asking and answer-evaluation skills
• Linking metacognitive skills to better AI usage