FaLCon: Facet-Anchored Retrieval with Late Consensus for Sim2Real Text-Based Person Anomaly Search

2026-08-10

AI Summary

This work addresses the challenge of fine-grained matching in text-based pedestrian anomaly retrieval under synthetic-to-real (Sim2Real) scenarios. To this end, the authors propose an anchor-constrained coarse-to-fine retrieval framework that leverages multi-facet semantic decomposition and calibrated fusion. The approach integrates a heterogeneous vision-language retriever, a Qwen3-based reranker, and an anomaly-aware cloze-style verification module, complemented by an uncertainty-gated consensus mechanism operating over a small candidate pool to enable efficient fine-grained semantic reasoning. Innovatively, semantic facets serve as anchor constraints to jointly optimize recall and computational efficiency. Evaluated on the PAB benchmark, the method achieves 95.41% mAP@10, 94.44% R@1, and 99.09% R@5, significantly outperforming existing single-backbone models.

Abstract

Text-based person anomaly search requires retrieving real-world pedestrian images from detailed natural-language descriptions using models trained primarily on synthetic data. This Sim2Real setting is particularly challenging because visually similar candidates may differ only in subtle actions, object interactions, or appearance attributes, while applying multimodal large language models to the entire gallery is computationally expensive. We propose an anchor-constrained coarse-to-fine retrieval framework that combines global semantic matching with fine-grained verification. First, each query is represented by its original caption, a structured concatenation, and several semantic facets. Heterogeneous vision-language retrievers are then integrated through robust per-query score calibration and soft claim-aware fusion. Full and concatenated captions serve as anchors to preserve candidate recall, whereas appearance, action, and object facets provide bounded corrective evidence. The resulting candidate pool is further refined by a discriminative Qwen3 reranker and two complementary semantic verification modules based on anomaly-aware cloze completion and multi-agent evidence reasoning. Finally, an uncertainty-gated consensus module adaptively reweights the three experts on ambiguous queries. Experiments on the PAB benchmark show that the proposed soft claim-aware retrieval achieves 86.44% mAP@10, substantially outperforming individual retrieval backbones. The complete framework further improves performance to 95.41% mAP@10, 94.44% R@1, and 99.09% R@5. These results demonstrate that preserving strong global retrieval while restricting expensive semantic reasoning to a small candidate pool is effective for fine-grained Sim2Real person anomaly search. Our code will be available on Github.

Problem

• text-based person anomaly search
• Sim2Real
• fine-grained retrieval
• vision-language retrieval
• anomaly detection

Innovation

• Sim2Real retrieval
• facet-based semantic decomposition
• soft claim-aware fusion
• uncertainty-gated consensus
• text-based person anomaly search