Atomic Learning Objectives Labeling: A High-Resolution Approach for Physics Education

2024-12-13

arXiv.org

AI Summary

Introductory university physics courses often employ coarse-grained learning objectives (LOs), hindering precise instructional diagnosis and adaptive support. Method: We propose a high-resolution, atomic-level LO framework, formalized via subject–predicate–object structures to precisely encode cognitive operations and conceptual elements required for problem solving. Leveraging cognitive task analysis, we design a structured prompt engineering pipeline to automatically annotate 131 problems—spanning nine core chapters—with 1–8 atomic LOs per item, using LLaMA-3 and GPT-4. A multidimensional evaluation metric suite is introduced, and inter-rater agreement with expert annotations reaches 76.3%. Contribution/Results: This work establishes the first systematic modeling framework for atomic LOs, empirically characterizing LLMs’ capabilities and limitations in educational objective inference. It delivers an interpretable, traceable “learning GPS” infrastructure for intelligent tutoring systems.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Structured Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This paper introduces a novel approach to create a high-resolution"map"for physics learning: an"atomic"learning objectives (LOs) system designed to capture detailed cognitive processes and concepts required for problem solving in a college-level introductory physics course. Our method leverages Large Language Models (LLMs) for automated labeling of physics questions and introduces a comprehensive set of metrics to evaluate the quality of the labeling outcomes. The atomic LO system, covering nine chapters of an introductory physics course, uses a"subject-verb-object'' structure to represent specific cognitive processes. We apply this system to 131 questions from expert-curated question banks and the OpenStax University Physics textbook. Each question is labeled with 1-8 atomic LOs across three chapters. Through extensive experiments using various prompting strategies and LLMs, we compare automated LOs labeling results against human expert labeling. Our analysis reveals both the strengths and limitations of LLMs, providing insight into LLMs reasoning processes for labeling LOs and identifying areas for improvement in LOs system design. Our work contributes to the field of learning analytics by proposing a more granular approach to mapping learning objectives with questions. Our findings have significant implications for the development of intelligent tutoring systems and personalized learning pathways in STEM education, paving the way for more effective"learning GPS'' systems.

Problem

• Automated labeling of physics questions
• Evaluation of labeling quality metrics
• Development of intelligent tutoring systems

Innovation

• Uses Large Language Models for labeling
• Introduces atomic learning objectives system
• Evaluates labeling with comprehensive metrics