Comparing RAG and GraphRAG for Page-Level Retrieval Question Answering on Math Textbook

2025-09-20

AI Summary

Large language models (LLMs) exhibit insufficient alignment with domain-specific curricular knowledge in page-level question answering over undergraduate mathematics textbooks. Method: We construct a benchmark dataset of 477 question-answer pairs and conduct the first systematic comparison between embedding-based RAG and knowledge-graph-enhanced GraphRAG in this setting. GraphRAG structures retrieval by modeling semantic relationships among mathematical concepts, while both approaches evaluate answer quality via F1 score; an LLM-based re-ranking step is further introduced for optimization. Contribution/Results: Embedding-based RAG significantly outperforms GraphRAG in both page retrieval accuracy and answer quality. GraphRAG’s entity-driven mechanism induces over-retrieval and redundancy, and LLM re-ranking fails to improve performance—instead exacerbating hallucination and degrading outcomes. This work empirically delineates the applicability boundaries of structured retrieval in educational contexts and provides critical methodological insights and cautions for curriculum-knowledge alignment.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Technology-enhanced learning environments often help students retrieve relevant learning content for questions arising during self-paced study. Large language models (LLMs) have emerged as novel aids for information retrieval during learning. While LLMs are effective for general-purpose question-answering, they typically lack alignment with the domain knowledge of specific course materials such as textbooks and slides. We investigate Retrieval-Augmented Generation (RAG) and GraphRAG, a knowledge graph-enhanced RAG approach, for page-level question answering in an undergraduate mathematics textbook. While RAG has been effective for retrieving discrete, contextually relevant passages, GraphRAG may excel in modeling interconnected concepts and hierarchical knowledge structures. We curate a dataset of 477 question-answer pairs, each tied to a distinct textbook page. We then compare the standard embedding-based RAG methods to GraphRAG for evaluating both retrieval accuracy-whether the correct page is retrieved-and generated answer quality via F1 scores. Our findings show that embedding-based RAG achieves higher retrieval accuracy and better F1 scores compared to GraphRAG, which tends to retrieve excessive and sometimes irrelevant content due to its entity-based structure. We also explored re-ranking the retrieved pages with LLM and observed mixed results, including performance drop and hallucinations when dealing with larger context windows. Overall, this study highlights both the promises and challenges of page-level retrieval systems in educational contexts, emphasizing the need for more refined retrieval methods to build reliable AI tutoring solutions in providing reference page numbers.

Problem

• Evaluating RAG versus GraphRAG for textbook page retrieval accuracy
• Assessing question-answering performance on domain-specific mathematics content
• Comparing embedding-based and knowledge graph-enhanced retrieval methods

Innovation

• RAG uses embedding-based retrieval for passages
• GraphRAG employs knowledge graphs for interconnected concepts
• LLM re-ranking tested but caused performance drop