Probe, Don't Prompt: A Hidden-State Probe for Metadata Filtering in Multi-Meta-RAG

2026-07-04

AI Summary

Current large language models rely on prompting to extract metadata for multi-hop question answering retrieval, suffering from high computational costs, uncontrollable outputs, and an inability to reject empty queries. This work proposes a deterministic probing method based on shallow hidden states of small-scale open-source language models, which accurately extracts metadata from a predefined vocabulary using mean pooling and a lightweight linear classification head—requiring only a few initial layers of forward propagation. The approach enables rejection of empty queries, enforces strictly constrained outputs, and eliminates the need for external API calls. Evaluated on 2,556 queries, it achieves a set-wise exact match accuracy of 90.9%, substantially outperforming GPT-3.5 (80.9%) and a substring baseline (88.0%), with a 135M-parameter model matching the performance of a 1.5B-parameter counterpart.

Abstract

Multi-Meta-RAG improves retrieval for multi-hop question answering by filtering a vector store on metadata (the news source) that it extracts from each query by prompting gpt-3.5-turbo. We show this proprietary, free-form extractor can be replaced by a local, deterministic probe trained on the hidden states of a small open-source language model. On all 2556 MultiHop-RAG queries the probe reaches 90.9% set-exact accuracy against 88.0% for a model-free substring baseline and 80.9% for GPT-3.5, a margin that comes entirely from null queries, on which GPT-3.5 never abstains; on non-null queries all three stay within about a point. Because the probe's output space is exactly the fixed 49-source vocabulary, it cannot drift outside the allow-list as the prompted model does. Three design choices make it work: selecting a shallow layer, mean pooling, and class-imbalance-aware multi-label training over the long tail of sources. A 135M-parameter model lands within ~1.5 points of a 1.5B one, so the filter is cheap to output: a partial forward pass through the first few layers plus one linear head, with no API. The code is available at https://github.com/mxpoliakov/Multi-Meta-RAG.

Problem

• metadata filtering
• multi-hop question answering
• hidden-state probing
• RAG
• language model probing

Innovation

• hidden-state probing
• metadata filtering
• Multi-Meta-RAG
• deterministic probe
• class-imbalance-aware training