MCompassRAG: Topic Metadata as a Semantic Compass for Paragraph-Level Retrieval

2026-06-16

AI Summary

This work addresses the challenge in retrieval-augmented generation (RAG) systems where chunking granularity often entails a trade-off between retrieval accuracy and efficiency. The authors propose a metadata-guided retrieval framework that, for the first time, integrates topic-level metadata as a semantic compass into the embedding space to enrich paragraph representations. They further employ a large language model to train a lightweight retriever via teacher distillation, enabling topic-aware retrieval without additional LLM invocations. Evaluated on six challenging retrieval benchmarks, the method achieves an average 8.24% improvement in information efficiency while maintaining latency below one-fifth of the current strongest efficient RAG baseline, significantly enhancing inference efficiency without compromising retrieval accuracy.

Abstract

Retrieval-augmented generation (RAG) systems depend critically on how documents are chunked and searched. Fine-grained chunks can improve retrieval precision but expand the search space, increasing latency and cost; larger chunks reduce the number of candidates but make dense similarity less reliable, as the representation for each chunk mixes multiple topics and introduces more semantic noise. This trade-off becomes especially limiting in deep research tasks, where retrieval must be both fast and precise across large, heterogeneous corpora. We introduce MCompassRAG, a metadata-guided retrieval framework that uses topic-level signals as a semantic compass for selecting relevant evidence. Instead of relying only on cosine similarity between queries and noisy chunk embeddings, MCompassRAG enriches chunk representations with topic metadata in the same embedding space and trains a lightweight retriever through LLM-teacher distillation. At inference time, MCompassRAG performs topic-aware retrieval without additional LLM calls, improving both efficiency and evidence quality. Across six complex retrieval benchmarks, MCompassRAG improves information efficiency (IE) by 8.24% on average with over 5 times lower latency than the strongest efficient RAG baselines. Code is available on https://github.com/AmirAbaskohi/MCompassRAG.

Problem

• Retrieval-augmented generation
• document chunking
• semantic noise
• topic heterogeneity
• retrieval efficiency

Innovation

• topic metadata
• semantic compass
• retrieval-augmented generation
• LLM distillation
• paragraph-level retrieval