RAPT: Retrieval-Augmented Post-hoc Thresholding for Multi-Label Classification

2026-05-15

AI Summary

This work addresses the limitations of global thresholds in multi-label classification, which struggle with OCR noise, label imbalance, instance-dependent label counts, asymmetric error costs, and evolving document formats. To overcome these challenges without retraining or fine-tuning the base classifier, the authors propose RAPT—a deployment-oriented, retrieval-augmented post-processing method that dynamically adjusts label selection thresholds by retrieving threshold decisions from similar historical documents. As the first approach to integrate retrieval augmentation into multi-label post-processing, RAPT enables model-agnostic, adaptive threshold calibration. Experiments demonstrate its consistent superiority over static thresholding across an industrial dataset and six public benchmarks, achieving a Macro-F1 of 0.87 in real-world settings, while being 115× faster at inference and using 13.5× less GPU memory than few-shot LLM-based alternatives.

Abstract

Industrial multi-label document understanding pipelines score candidate labels and threshold or rank them to form a label set per document. This early selection step directly affects the accuracy of downstream information extraction from the document, as well as the associated verification effort. In practice, OCR noise, label imbalance, instance-dependent label cardinality, and asymmetric error costs make global score thresholds brittle and hard to maintain as document formats evolve. We present RAPT, a deployment-oriented retrieval-augmented score thresholding wrapper, applied post-hoc to improve label set selection without retraining the underlying classifier. RAPT is a model-agnostic wrapper: any predictor that provides document representations for similarity search and per label confidence scores can be used, including metric learning encoders and fine-tuned transformer classifiers. For each query document, given a classifier's score vector, RAPT retrieves similar document thresholding situations (cases) and adapts the query's label set selection threshold using their outcomes. The adaptation selects the final label set by locally aggregating neighbour solutions (e.g. average label count, cutoff calibration). Evaluation compared multi-label classifiers (metric learners and transformers) combined with RAPT against global and label-wise thresholding baselines, and against few-shot LLMs. Across an industrial dataset and six public benchmarks, RAPT consistently outperformed global and label-wise static thresholding baselines. In the industrial setting, RAPT achieved its best predictive performance with metric learners, reaching 0.87 Macro-F1, while fine-tuned transformer variants on average achieved 0.775 Macro-F1, outperforming fewshot LLM baselines (K = 5) by 2x and requiring at least 115x less inference time and 13.5x less GPU memory.

Problem

• multi-label classification
• thresholding
• document understanding
• label imbalance
• OCR noise

Innovation

• retrieval-augmented thresholding
• multi-label classification
• post-hoc calibration
• model-agnostic wrapper
• instance-adaptive thresholding