Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method

2026-06-06

AI Summary

This work addresses the high cost of directly querying large language models (LLMs) for semantic document filtering at scale, a challenge exacerbated by existing cascade methods that suffer from limited representation diversity, neglect of fine-grained evidence, and over-calibration. To overcome these limitations, the authors propose an adaptive two-stage filtering framework that integrates model-free clustering with online proxy model training, sharing oracle calls between stages. The approach introduces a token-aware hybrid model, soft-label supervision, and a sparsity-aware calibration mechanism, while unifying multiple cascade strategies into a dynamically adaptable system for the first time. Experiments on three datasets containing tens of thousands of documents demonstrate that the method achieves 95% query success rate at 90% accuracy, outperforming the best baseline by 1.6–2.0× in speed and revealing a potential optimization margin of 4–20×.

Abstract

Evaluating a natural-language yes/no predicate over a document corpus under an accuracy target - the semantic filter - is a cornerstone of LLM-based data processing. Calling the LLM on every document (the oracle) is prohibitive, so cascades pair the oracle with a fast proxy. As deployed today, they leave four limitations on the table. (1) Each cascade family - model-free clustering, prebuilt small-LLM proxies, online-trained proxies - commits to a single representation and pipeline, and wins on only a narrow query regime. (2) The strongest online proxy invests in a custom training scheme on a bi-encoder over dense embeddings, missing the token-level evidence richer predicates require. (3) The proxy is trained against binary yes/no labels, wasting the LLM's per-document confidence at the boundary documents it most needs to learn. (4) Existing calibrations add a uniform safety margin, conflating genuine proxy uncertainty with small-sample noise and inflating cascade cost. We address these by (1) composing families adaptively - model-free clustering first, online proxy only when needed, with oracle calls shared across phases; (2) replacing the cosine bi-encoder with a hybrid of off-the-shelf token-aware models; (3) training the proxy with the oracle's per-document confidence as a soft label; and (4) a calibration that adds the safety margin only where the labeled sample is sparse. We are also the first to use the oracle's per-document confidence for three purposes: a query-level difficulty compass, a lower bound on the minimum oracle calls any proxy-based cascade can make, and the proxy's soft training label. At a 90% accuracy target on three 10K-document corpora, our methods are 1.6-2.0x faster than the best prior method per corpus and meet the target on 95% of queries; the BER-derived lower bound indicates a further ~4-20x of headroom for future work.

Problem

• semantic filtering
• large language models
• cascade methods
• proxy models
• confidence calibration

Innovation

• semantic filtering
• adaptive cascade
• token-aware proxy
• soft-label training
• confidence calibration