Unlocking Scientific Concepts: How Effective Are LLM-Generated Analogies for Student Understanding and Classroom Practice?

2025-02-24

AI Summary

Prior work lacks empirical evidence on the pedagogical efficacy and implementation mechanisms of large language model (LLM)-generated analogies in secondary science education. Method: We conducted a two-phase controlled experiment and classroom-based field study to investigate how LLM-generated analogies impact conceptual learning in high school biology and physics, identifying critical instructional requirements. Contribution/Results: We empirically demonstrate that LLM-generated analogies require deliberate teacher scaffolding—without guidance, they induce student overconfidence. Based on this finding, we propose the “Generate–Refine–Elicit” pedagogical闭环 and develop a practical system enabling teachers to autonomously customize and iteratively optimize analogies through interactive refinement. Results show statistically significant improvements in biology concept understanding (p < 0.01) and a 23% gain in assignment performance. All participating teachers successfully refined generated analogies independently and created novel, domain-specific teaching analogies. This work bridges dual gaps in LLM educational applications: human-AI co-design frameworks and rigorous empirical validation.

Technology Category

• Cognitive Modeling & Cognitive Systems: Analogy
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Teaching scientific concepts is essential but challenging, and analogies help students connect new concepts to familiar ideas. Advancements in large language models (LLMs) enable generating analogies, yet their effectiveness in education remains underexplored. In this paper, we first conducted a two-stage study involving high school students and teachers to assess the effectiveness of LLM-generated analogies in biology and physics through a controlled in-class test and a classroom field study. Test results suggested that LLM-generated analogies could enhance student understanding particularly in biology, but require teachers' guidance to prevent over-reliance and overconfidence. Classroom experiments suggested that teachers could refine LLM-generated analogies to their satisfaction and inspire new analogies from generated ones, encouraged by positive classroom feedback and homework performance boosts. Based on findings, we developed and evaluated a practical system to help teachers generate and refine teaching analogies. We discussed future directions for developing and evaluating LLM-supported teaching and learning by analogy.

Problem

• Assessing LLM-generated analogies' educational effectiveness
• Exploring teachers' role in refining LLM analogies
• Developing a system for generating teaching analogies

Innovation

• LLM-generated analogies for teaching
• Teacher-guided refinement of analogies
• Practical system for analogy generation