Evaluating Chunking Strategies for Retrieval-Augmented Generation on Academic Texts

2026-07-02

AI Summary

This study investigates optimal text chunking strategies for enhancing the response quality of Retrieval-Augmented Generation (RAG) systems when applied to structurally complex academic papers. We systematically compare semantic clustering, fixed-length, and recursive chunking approaches, evaluating output faithfulness and relevance using the RAGAs framework. To our knowledge, this is the first empirical comparison of multiple chunking strategies on long-form scholarly texts. Our findings indicate that semantic clustering does not significantly outperform simpler methods, and that question type—generic versus document-specific—substantially influences system performance. Furthermore, we identify limitations in the reliability of RAGAs’ faithfulness metric for such tasks, suggesting a need for more robust evaluation measures in academic RAG applications.

Abstract

Retrieval-Augmented Generation (RAG) systems use the question-answering capabilities of Large Language Models (LLMs) to access information outside their parameters. We evaluate if cluster-based semantic chunking improves retrieval and answer quality compared to fixed-size and recursive chunking evaluating on long, structured academic theses using the Retrieval Augmented Generation Assessment (RAGAs) framework. RAGAs based faithfulness shows limited reliability in this setup. Performance on fixed versus document specific questions varied substantially, likely related to the formatting of documents and preprocessing. Under the tested configuration, cluster-based chunking did not outperform simpler strategies.

Problem

• Retrieval-Augmented Generation
• chunking strategies
• academic texts
• semantic chunking
• RAG evaluation

Innovation

• cluster-based chunking
• retrieval-augmented generation
• semantic chunking
• RAG evaluation
• academic text processing