CS-Guide: Leveraging LLMs and Student Reflections to Provide Frequent, Scalable Academic Monitoring Feedback to Computer Science Students

2025-12-22

AI Summary

Severe faculty-to-student ratio imbalances in Computer Science programs—often exceeding 350:1, far above the recommended 250:1—lead to delayed academic monitoring and a lack of personalized feedback. Method: This study proposes the first real-time academic support framework integrating learning analytics, domain-adapted large language model (LLM)-based semantic understanding, and an executable intervention-feedback loop. It processes weekly structured assessment data and unstructured reflective logs, leveraging fine-tuned LLMs to extract fine-grained, pedagogically meaningful features and dynamically trigger timely academic interventions or mental health referrals. Contribution/Results: It introduces the first high-temporal-resolution, domain-specific academic early-warning and response system grounded in student-generated textual narratives. Evaluated on ~20,000 real-world records over four years, the framework achieves a 97% F1 score for intervention recommendations for first-year students, significantly improving coverage, consistency, and responsiveness of academic advising systems.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Computer Science (CS) departments often serve large student populations, making timely academic monitoring and personalized feedback difficult. While the recommended counselor-to-student ratio is 250:1, it often exceeds 350:1 in practice, leading to delays in support and interventions. We present CS-Guide, which leverages Large Language Models (LLMs) to deliver scalable, frequent academic feedback. Weekly, students interact with CS-Guide through self-reported grades and reflective journal entries, from which CS-Guide extracts quantitative and qualitative features and triggers tailored interventions (e.g., academic support, health and wellness referrals). Thus, CS-Guide uniquely integrates learning analytics, LLMs, and actionable interventions using both structured and unstructured student-generated data. We evaluated CS-Guide on a four-year, ~20K-entry longitudinal dataset, and it achieved up to a 97% F1 score in recommending interventions for first-year students. This shows that CS-Guide can enhance advising systems with scalable, consistent, timely, and domain-specific feedback.

Problem

• Provide scalable academic feedback to large CS student populations
• Address high counselor-to-student ratios with timely interventions
• Integrate LLMs and student reflections for personalized monitoring

Innovation

• LLMs analyze student reflections for feedback
• Combines structured and unstructured data for interventions
• Scalable academic monitoring with high accuracy predictions