Improving Automated Feedback Systems for Tutor Training in Low-Resource Scenarios through Data Augmentation

2025-01-16

AI Summary

In low-resource settings for novice mentor training, automated feedback systems suffer from limited performance due to scarcity of expert-annotated data. To address this, we propose a synthetic-data-driven approach for praise behavior recognition. Specifically, we leverage high-quality synthetic data generated by GPT-4o to augment fine-tuning of GPT-3.5 on a small set of real human annotations, employing BIO-style sequence labeling and an expert-in-the-loop verification framework. Our method significantly improves cross-praise-type generalization: under identical annotation budgets, it achieves a 12.7% absolute gain in F1 score on unseen praise categories. Results empirically validate the efficacy of large language model–generated synthetic data for low-resource educational AI tasks, establishing a scalable, cost-effective paradigm for intelligent teaching feedback systems.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Tutoring is an effective instructional method for enhancing student learning, yet its success relies on the skill and experience of the tutors. This reliance presents challenges for the widespread implementation of tutoring, particularly in training novice tutors. To support tutor training programs, real-time automated feedback systems are essential for efficiently training large numbers of tutors. Lin et al.'s previous study employed Generative Pre-Trained Transformers (GPT) for sequence labeling to identify desirable and undesirable praise components in a tutor training dataset, providing explanatory feedback. However, this approach requires a significant amount of labeled data for fine-tuning, which is both labor-intensive and dependent on expert input. To address the challenges associated with extensive data labeling, the current study explores the use of prompting more advanced GPT models like GPT-4o to generate synthetic datasets for augmenting labeled response data, followed by fine-tuning a GPT-3.5 model. Our results demonstrate that our data augmentation approach generalizes effectively to identify other types of praise, compared to the same model fine-tuned without augmentation. These findings suggest that for data-intensive tasks, synthetic data generated through GPT model prompting can substantially enhance fine-tuned model performance in low-resource scenarios.

Problem

• Limited Resources
• Expert Annotation Shortage
• Automated Feedback Systems

Innovation

• GPT-generated synthetic data
• Enhanced training method
• Feedback system accuracy improvement