Answering Students'Questions on Course Forums Using Multiple Chain-of-Thought Reasoning and Finetuning RAG-Enabled LLM

2025-11-13

AI Summary

To address the surge in student queries, delayed instructor responses, and redundant answering burdens in course forums, this paper proposes an education-oriented RAG-enhanced question-answering system. Methodologically, it integrates Retrieval-Augmented Generation (RAG) with Multi-Chain-of-Thought reasoning: a vector retrieval module is built upon a local course knowledge base, and an open-source large language model is domain-finetuned to mitigate hallucination and improve answer accuracy. Key contributions include: (i) embedding multi-step reasoning into the RAG pipeline to enhance logical coherence; and (ii) devising a lightweight course-knowledge-adaptive fine-tuning strategy. Experiments on the HotpotQA benchmark and real-world course QA tasks demonstrate that the system achieves significantly higher accuracy than baseline models, effectively enabling high-concurrency, low-latency automated teaching assistance.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Argumentation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Community question answering

Abstract

The course forums are increasingly significant and play vital role in facilitating student discussions and answering their questions related to the course. It provides a platform for students to post their questions related to the content and admin issues related to the course. However, there are several challenges due to the increase in the number of students enrolled in the course. The primary challenge is that students'queries cannot be responded immediately and the instructors have to face lots of repetitive questions. To mitigate these issues, we propose a question answering system based on large language model with retrieval augmented generation (RAG) method. This work focuses on designing a question answering system with open source Large Language Model (LLM) and fine-tuning it on the relevant course dataset. To further improve the performance, we use a local knowledge base and applied RAG method to retrieve relevant documents relevant to students'queries, where the local knowledge base contains all the course content. To mitigate the hallucination of LLMs, We also integrate it with multi chain-of-thought reasoning to overcome the challenge of hallucination in LLMs. In this work, we experiment fine-tuned LLM with RAG method on the HotpotQA dataset. The experimental results demonstrate that the fine-tuned LLM with RAG method has a strong performance on question answering task.

Problem

• Automating responses to repetitive student questions on course forums
• Reducing instructor workload by providing immediate query resolution
• Mitigating LLM hallucinations through multi-step reasoning and fine-tuning

Innovation

• Fine-tuned open source LLM on course dataset
• Applied RAG method with local knowledge base
• Integrated multi chain-of-thought reasoning for accuracy