Using Learning Progressions to Guide AI Feedback for Science Learning

2026-03-03

AI Summary

This study addresses the high cost and limited generalizability of traditional AI feedback systems that rely on expert-crafted rubrics. It proposes a novel approach that leverages Learning Progressions (LPs) to automatically generate scoring rubrics, which in turn guide large language models in delivering formative feedback on middle school students’ scientific explanations. The work presents the first empirical validation demonstrating that LP-derived rubrics can effectively substitute for human-authored ones, with no statistically significant differences observed between LP-driven and expert-driven AI feedback across key dimensions—clarity, relevance, engagement, and reflectiveness. By integrating learning progression theory, automated rubric generation, and large language model capabilities, this method establishes a scalable and high-quality AI feedback mechanism for educational contexts.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Generative artificial intelligence (AI) offers scalable support for formative feedback, yet most AI-generated feedback relies on task-specific rubrics authored by domain experts. While effective, rubric authoring is time-consuming and limits scalability across instructional contexts. Learning progressions (LP) provide a theoretically grounded representation of students' developing understanding and may offer an alternative solution. This study examines whether an LP-driven rubric generation pipeline can produce AI-generated feedback comparable in quality to feedback guided by expert-authored task rubrics. We analyzed AI-generated feedback for written scientific explanations produced by 207 middle school students in a chemistry task. Two pipelines were compared: (a) feedback guided by a human expert-designed, task-specific rubric, and (b) feedback guided by a task-specific rubric automatically derived from a learning progression prior to grading and feedback generation. Two human coders evaluated feedback quality using a multi-dimensional rubric assessing Clarity, Accuracy, Relevance, Engagement and Motivation, and Reflectiveness (10 sub-dimensions). Inter-rater reliability was high, with percent agreement ranging from 89% to 100% and Cohen's kappa values for estimable dimensions (kappa = .66 to .88). Paired t-tests revealed no statistically significant differences between the two pipelines for Clarity (t1 = 0.00, p1 = 1.000; t2 = 0.84, p2 = .399), Relevance (t1 = 0.28, p1 = .782; t2 = -0.58, p2 = .565), Engagement and Motivation (t1 = 0.50, p1 = .618; t2 = -0.58, p2 = .565), or Reflectiveness (t = -0.45, p = .656). These findings suggest that the LP-driven rubric pipeline can serve as an alternative solution.

Problem

• generative AI
• formative feedback
• learning progressions
• rubric authoring
• science learning

Innovation

• Learning Progressions
• AI-generated feedback
• rubric generation
• formative assessment
• science education