Action-Aligned Retrieval with Pairwise Multimodal Reranking for Text-Based Person Anomaly Search

2026-08-24

AI Summary

本文针对文本驱动的人体异常行为搜索问题，提出ActPair框架，通过动作对齐检索与成对多模态重排序结合的方法来解决现有方法难以捕捉上下文条件行为的问题。

Abstract

Text-based person anomaly search requires distinguishing individuals based on fine-grained, context-dependent behaviors rather than mere appearance. Existing methods struggle to capture these context-conditioned actions, frequently relying on isolated skeletal geometry, discarding raw query details during reformulation, or utilizing absolute pointwise scoring for multimodal verification. To address these limitations, we propose \textbf{ActPair}, a unified three-stage coarse-to-fine framework that combines action-aligned retrieval with pairwise multimodal reranking to bridge the pose-semantic gap. First, we fine-tune a vision-language model (VLM) with an action-aligned multi-task objective that encourages the representations to encode action-discriminative semantics. Second, we perform parallel late-fusion retrieval using the original query and a large language model (LLM)-generated context-grounded rewrite, retaining complementary details from both semantic views. Finally, we propose an efficient off-the-shelf reranking module that leverages a pivot-promote algorithm to perform direct pairwise visual comparisons, mitigating residual spatial and compositional ambiguities without the prohibitive inference costs of exhaustive evaluation. Extensive experiments demonstrate that our framework achieves the best results among the compared methods on the Pedestrian Anomaly Behavior (PAB) public test and transfers effectively to an unseen, non-anomaly-specific dataset.

Problem

• Text-based person anomaly search
• context-dependent behaviors
• multimodal verification

Innovation

• action-aligned retrieval
• pairwise multimodal reranking
• pivot-promote algorithm