PEMUTA: Pedagogically-Enriched Multi-Granular Undergraduate Thesis Assessment

2025-07-25

AI Summary

Current LLM-based automated evaluation of undergraduate thesis essays (UGTEs) yields only a single holistic score, failing to capture pedagogically critical dimensions—such as structural coherence, alignment with learning objectives, and multifaceted academic competencies. To address this, we propose SLOWPR, a teaching-oriented, fine-grained assessment framework. SLOWPR is the first to integrate Vygotsky’s Zone of Proximal Development and Bloom’s Taxonomy into prompt engineering, establishing a six-dimensional rubric covering structure, logic, domain knowledge, writing quality, reflective depth, and academic integrity. Leveraging hierarchical role-playing and few-shot in-context learning, it achieves alignment between large language models and pedagogical expertise without model fine-tuning. Experiments demonstrate strong inter-rater agreement between SLOWPR and human experts across all dimensions (mean Cohen’s κ = 0.82), significantly enhancing both the pedagogical validity and explanatory granularity of automated thesis evaluation.

Technology Category

• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Machine Learning: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The undergraduate thesis (UGTE) plays an indispensable role in assessing a student's cumulative academic development throughout their college years. Although large language models (LLMs) have advanced education intelligence, they typically focus on holistic assessment with only one single evaluation score, but ignore the intricate nuances across multifaceted criteria, limiting their ability to reflect structural criteria, pedagogical objectives, and diverse academic competencies. Meanwhile, pedagogical theories have long informed manual UGTE evaluation through multi-dimensional assessment of cognitive development, disciplinary thinking, and academic performance, yet remain underutilized in automated settings. Motivated by the research gap, we pioneer PEMUTA, a pedagogically-enriched framework that effectively activates domain-specific knowledge from LLMs for multi-granular UGTE assessment. Guided by Vygotsky's theory and Bloom's Taxonomy, PEMUTA incorporates a hierarchical prompting scheme that evaluates UGTEs across six fine-grained dimensions: Structure, Logic, Originality, Writing, Proficiency, and Rigor (SLOWPR), followed by holistic synthesis. Two in-context learning techniques, ie, few-shot prompting and role-play prompting, are also incorporated to further enhance alignment with expert judgments without fine-tuning. We curate a dataset of authentic UGTEs with expert-provided SLOWPR-aligned annotations to support multi-granular UGTE assessment. Extensive experiments demonstrate that PEMUTA achieves strong alignment with expert evaluations, and exhibits strong potential for fine-grained, pedagogically-informed UGTE evaluations.

Problem

• Addresses lack of multi-granular assessment in UGTE evaluations
• Integrates pedagogical theories into automated LLM-based assessment
• Enhances alignment with expert judgments using hierarchical prompting

Innovation

• Hierarchical prompting scheme for multi-granular assessment
• Incorporates few-shot and role-play prompting techniques
• Leverages Vygotsky's theory and Bloom's Taxonomy