Annotation Guidelines-Based Knowledge Augmentation: Towards Enhancing Large Language Models for Educational Text Classification

2024-06-03

arXiv.org

AI Summary

This study addresses the suboptimal zero-shot and few-shot performance of large language models (LLMs) on Learning Engagement Classification (LEC)—a fine-grained educational text classification task spanning behavioral, emotional, and cognitive dimensions across six engagement categories. We propose the novel Annotation-Guided Knowledge Augmentation (AGKA) paradigm: GPT-4 automatically parses label definitions to construct structured prior knowledge, integrated with random undersampling to optimize few-shot prompting. We also introduce the first comprehensive, multi-dimensional, cross-scenario LEC benchmark for systematic evaluation. Experiments show that GPT-4+AGKA outperforms fully fine-tuned BERT and RoBERTa on binary classification; remarkably, Llama 3 70B+AGKA achieves comparable performance to GPT-4+AGKA. Our analysis identifies critical bottlenecks in LLMs’ ability to distinguish semantically similar labels and perform multi-class fine-grained classification. The AGKA framework provides a transferable methodology for zero-shot semantic understanding in educational AI.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Various machine learning approaches have gained significant popularity for the automated classification of educational text to identify indicators of learning engagement -- i.e. learning engagement classification (LEC). LEC can offer comprehensive insights into human learning processes, attracting significant interest from diverse research communities, including Natural Language Processing (NLP), Learning Analytics, and Educational Data Mining. Recently, Large Language Models (LLMs), such as ChatGPT, have demonstrated remarkable performance in various NLP tasks. However, their comprehensive evaluation and improvement approaches in LEC tasks have not been thoroughly investigated. In this study, we propose the Annotation Guidelines-based Knowledge Augmentation (AGKA) approach to improve LLMs. AGKA employs GPT 4.0 to retrieve label definition knowledge from annotation guidelines, and then applies the random under-sampler to select a few typical examples. Subsequently, we conduct a systematic evaluation benchmark of LEC, which includes six LEC datasets covering behavior classification (question and urgency level), emotion classification (binary and epistemic emotion), and cognition classification (opinion and cognitive presence). The study results demonstrate that AGKA can enhance non-fine-tuned LLMs, particularly GPT 4.0 and Llama 3 70B. GPT 4.0 with AGKA few-shot outperforms full-shot fine-tuned models such as BERT and RoBERTa on simple binary classification datasets. However, GPT 4.0 lags in multi-class tasks that require a deep understanding of complex semantic information. Notably, Llama 3 70B with AGKA is a promising combination based on open-source LLM, because its performance is on par with closed-source GPT 4.0 with AGKA. In addition, LLMs struggle to distinguish between labels with similar names in multi-class classification.

Problem

• Enhancing large language models for educational text classification tasks
• Improving LLMs' performance in learning engagement classification using annotation guidelines
• Addressing LLMs' limitations in multi-class educational classification with similar labels

Innovation

• AGKA augments LLMs using annotation guidelines knowledge
• AGKA selects typical examples via random under-sampling
• AGKA enhances non-fine-tuned LLMs for educational classification