🤖 AI Summary
This work addresses the inefficiencies of conventional text chunking in standard retrieval-augmented generation (RAG) systems, which often introduces redundancy, leading to excessive storage costs and degraded retrieval performance. To mitigate this, the authors propose a lightweight pre-index filtering mechanism that integrates semantic similarity, topic coherence, and named entity recognition to selectively prune redundant text chunks prior to indexing. Evaluated through a token-level precision, recall, and intersection-over-union framework, the approach reduces vector index size by 25%–36% while preserving retrieval quality comparable to that of the original system, thereby significantly enhancing the overall efficiency of RAG pipelines.
📝 Abstract
Standard Retrieval-Augmented Generation (RAG) chunking methods often create excessive redundancy, increasing storage costs and slowing retrieval. This study explores chunk filtering strategies, such as semantic, topic-based, and named-entity-based methods in order to reduce the indexed corpus while preserving retrieval quality. Experiments are conducted on multiple corpora. Retrieval performance is evaluated using a token-based framework based on precision, recall, and intersection-over-union metrics. Results indicate that entity-based filtering can reduce vector index size by approximately 25% to 36% while maintaining high retrieval quality close to the baseline. These findings suggest that redundancy introduced during chunking can be effectively reduced through lightweight filtering, improving the efficiency of retrieval-oriented components in RAG pipelines.