[CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregation

2026-05-25

AI Summary

This work addresses a key limitation of existing vision-language models such as CLIP, which rely on the [CLS] token for global representation and struggle to effectively model multiple co-occurring objects—including their scales, contextual dependencies, and co-occurrence patterns—in multi-label image recognition. To overcome this, the authors propose the Patch-level Inference with Adaptive Aggregation (PIAA) framework, which, without any training or parameter fine-tuning, enhances performance through patch-level inference and adaptive score aggregation. Specifically, PIAA enriches patch representations to mitigate semantic entanglement and constructs an unsupervised visual classifier, while adaptively fusing patch-level scores to produce the final prediction. Evaluated on NUS-WIDE, the method achieves a mean average precision (mAP) gain of over 6% compared to strong baselines, delivering substantial improvements with minimal computational overhead.

Abstract

Vision-Language Models such as CLIP exhibit strong zero-shot recognition capability by aligning images with textual concepts, yet they often underperform on multi-label recognition where multiple objects co-exist. A key bottleneck is that the [CLS] token, as a single global visual representation, is insufficient to faithfully encode diverse targets with varying scales, contexts, and co-occurrence patterns. To address this limitation, we present a new multi-label image recognition framework, termed PIAA, which formulates prediction as Patch-level Inference followed by Adaptive Aggregation. Specifically, we first enhance patch-wise predictions from two complementary perspectives: (i) mitigating semantic entanglement in the visual encoder to obtain more discriminative patch representations, and (ii) learning an unsupervised visual classifier to narrow the vision-language modality gap. We then introduce an adaptive aggregation module that consolidates patch-level scores into the final multi-label prediction. Notably, the entire pipeline is fully training-free, requiring no gradient updates or parameter fine-tuning. Experiments show that our method achieves strong improvements with minimal extra computation, exceeding a 6% mAP gain on the challenging NUS-WIDE benchmark over representative baselines. Code is available at https://github.com/akang-wang/PIAA.

Problem

• multi-label recognition
• vision-language models
• CLS token
• patch-level inference
• visual representation

Innovation

• Patch-level Inference
• Adaptive Aggregation
• Multi-label Recognition
• Vision-Language Models
• Training-free