Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs

2025-01-10

AI Summary

This study addresses the high deployment cost and resource consumption of large language models (LLMs) in educational settings, focusing on automated multiple-choice question (MCQ) answering for programming courses. To this end, we propose a lightweight adaptation framework comprising three key components: (1) constructing the first open-source, programming-language-course-specific MCQ dataset; (2) performing textbook-guided supervised fine-tuning (SFT); and (3) applying 4-bit quantization for model compression. Experiments are conducted across the LLaMA-2 family (7B, 13B, and 70B variants). Results demonstrate that the textbook-driven 7B model achieves higher accuracy than the unadapted 70B general-purpose model, while reducing GPU memory usage by 85% during inference. This approach significantly improves cost-effectiveness for pedagogical applications and delivers an efficient, deployable lightweight AI solution tailored for resource-constrained educational environments.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

In education, the capability of generating human-like text of Large Language Models (LLMs) inspired work on how they can increase the efficiency of learning and teaching. We study the affordability of these models for educators and students by investigating how LLMs answer multiple-choice questions (MCQs) with respect to hardware constraints and refinement techniques. We explore this space by using generic pre-trained LLMs (the 7B, 13B, and 70B variants of LLaMA-2) to answer 162 undergraduate-level MCQs from a course on Programming Languages (PL) -- the MCQ dataset is a contribution of this work, which we make publicly available. Specifically, we dissect how different factors, such as using readily-available material -- (parts of) the course's textbook -- for fine-tuning and quantisation (to decrease resource usage) can change the accuracy of the responses. The main takeaway is that smaller textbook-based fine-tuned models outperform generic larger ones (whose pre-training requires conspicuous resources), making the usage of LLMs for answering MCQs resource- and material-wise affordable.

Problem

• Language Models
• Educational Applications
• Programming Courses

Innovation

• Cost-effective Language Models
• Educational Applications
• Resource-saving Techniques