Fine-Tuning Large Language Models for Codebook-Guided Coding of Students' Mathematics Metaphor Responses

2026-08-10

AI Summary

Manually coding students’ metaphorical responses in mathematics is challenging to scale due to semantic complexity. This study presents the first application of compact open-source large language models—specifically DeepSeek-R1 1.5B and Mistral 7B—fine-tuned via LoRA-based supervised adaptation to automatically perform two types of coding tasks, namely sentiment intensity and thematic framing, guided by a formal coding manual. Compared to prompt-engineered approaches using GPT-4o mini and GPT-5 mini, the fine-tuned open-source models achieve comparable or superior accuracy and operational stability. The proposed approach enables scalable, locally deployable, and privacy-preserving automated analysis of educational metaphors, offering a practical alternative to proprietary closed-source models.

Abstract

Student-generated metaphors about mathematics can reveal students' attitudes, beliefs, identities, and experiences, but human expert coding of these thematically and semantically complex open-ended responses is time-intensive and difficult to scale. This study examines whether LoRA-based supervised fine-tuning of large language models (LLMs) can improve their performance on codebook-guided coding tasks for student mathematics metaphors. We used a human-coded corpus of 2,265 Grade 6-8 responses to food- and animal-based metaphor prompts and instructed the LLMs to perform two coding tasks: valence-intensity coding to capture the direction and strength of students' affective orientations toward mathematics, and thematic coding to capture students' framings of mathematics as expressed through their metaphors. We compared two proprietary models, GPT-4o mini and GPT-5 mini, under prompt-only conditions with two open-weight models, DeepSeek-R1 1.5B and Mistral 7B, evaluated before and after fine-tuning. Results show that fine-tuning substantially improved the performance and run-to-run reliability of the open-weight models across both tasks relative to their base versions. The fine-tuned compact open-weight models became competitive with, and often outperformed, the proprietary prompt-only models. These findings suggest that compact open-weight LLMs can support scalable, locally controllable, and privacy-conscious AI-assisted measurement of students' metaphor responses in mathematics education.

Problem

• metaphor coding
• mathematics education
• student responses
• codebook-guided coding
• scalability

Innovation

• LoRA fine-tuning
• codebook-guided coding
• student metaphors
• open-weight LLMs
• mathematics education