SlideItRight: Using AI to Find Relevant Slides and Provide Feedback for Open-Ended Questions

2025-05-07

AI Summary

This study addresses the challenge of enhancing student learning outcomes on open-ended questions through multimodal AI feedback. We propose a novel educational feedback paradigm that integrates large language model (LLM)-generated textual explanations with cross-modal semantic retrieval of relevant course slides, yielding real-time, interpretable, and context-aligned text-plus-visual feedback. To our knowledge, this is the first work to embed slide-based knowledge retrieval into an AI feedback loop, enabling an end-to-end multimodal feedback system. A large-scale online crowdsourced experiment demonstrates statistically significant improvements in both learning gains and perceived feedback quality. Students positively rated the feedback’s personalization and content relevance; however, trust in AI-generated feedback remained lower than in human feedback, and slide comprehension imposed non-negligible cognitive load. Our work establishes a new, interpretable, and scalable pathway for intelligent educational feedback systems.

Technology Category

• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Feedback is important in supporting student learning. While various automated feedback systems have been implemented to make the feedback scalable, many existing solutions only focus on generating text-based feedback. As is indicated in the multimedia learning principle, learning with more modalities could help utilize more separate channels, reduce the cognitive load and facilitate students' learning. Hence, it is important to explore the potential of Artificial Intelligence (AI) in feedback generation from and to different modalities. Our study leverages Large Language Models (LLMs) for textual feedback with the supplementary guidance from other modality - relevant lecture slide retrieved from the slides hub. Through an online crowdsourcing study (N=91), this study investigates learning gains and student perceptions using a 2x2 design (i.e., human feedback vs. AI feedback and with vs. without relevant slide), evaluating the clarity, engagement, perceived effectiveness, and reliability) of AI-facilitated multimodal feedback. We observed significant pre-to-post learning gains across all conditions. However, the differences in these gains were not statistically significant between conditions. The post-survey revealed that students found the slide feedback helpful in their learning process, though they reported difficulty in understanding it. Regarding the AI-generated open-ended feedback, students considered it personalized and relevant to their responses, but they expressed lower trust in the AI feedback compared to human-generated feedback.

Problem

• AI generates multimodal feedback for student learning
• Evaluates effectiveness of slide-supplemented AI feedback
• Compares student trust in AI vs human feedback

Innovation

• Leveraging LLMs for textual feedback generation
• Retrieving relevant slides for supplementary guidance
• Evaluating multimodal feedback via crowdsourcing study