Compositional Caching for Training-free Open-vocabulary Attribute Detection

2025-03-24

AI Summary

Existing attribute detection methods rely on manual annotations and predefined attribute vocabularies, limiting their ability to support open-vocabulary and fine-grained descriptions (e.g., “matte deep navy blue”). To address this, we propose the first training-free, open-vocabulary attribute detection framework. Our method leverages large language models to generate diverse attribute–object pair prompts and constructs a soft-labeled attribute–object compatibility cache via web-scale image retrieval. Cross-modal similarity scores—computed by vision-language models—are used for dynamic, similarity-weighted aggregation, enabling zero-shot inference. Key innovations include a compositional caching mechanism and a similarity-driven soft-label aggregation strategy, which jointly overcome constraints on attribute granularity and vocabulary openness. Extensive experiments across multiple public benchmarks demonstrate that our approach significantly outperforms zero-shot and cache-based baselines, matching the performance of fully supervised methods requiring extensive annotations—thereby validating the efficacy of training-free paradigms for open-vocabulary attribute recognition.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Attribute detection is crucial for many computer vision tasks, as it enables systems to describe properties such as color, texture, and material. Current approaches often rely on labor-intensive annotation processes which are inherently limited: objects can be described at an arbitrary level of detail (e.g., color vs. color shades), leading to ambiguities when the annotators are not instructed carefully. Furthermore, they operate within a predefined set of attributes, reducing scalability and adaptability to unforeseen downstream applications. We present Compositional Caching (ComCa), a training-free method for open-vocabulary attribute detection that overcomes these constraints. ComCa requires only the list of target attributes and objects as input, using them to populate an auxiliary cache of images by leveraging web-scale databases and Large Language Models to determine attribute-object compatibility. To account for the compositional nature of attributes, cache images receive soft attribute labels. Those are aggregated at inference time based on the similarity between the input and cache images, refining the predictions of underlying Vision-Language Models (VLMs). Importantly, our approach is model-agnostic, compatible with various VLMs. Experiments on public datasets demonstrate that ComCa significantly outperforms zero-shot and cache-based baselines, competing with recent training-based methods, proving that a carefully designed training-free approach can successfully address open-vocabulary attribute detection.

Problem

• Detects open-vocabulary attributes without training
• Overcomes limitations of predefined attribute sets
• Leverages web data and LLMs for compatibility

Innovation

• Leverages web-scale databases and LLMs
• Uses soft attribute labels for cache images
• Model-agnostic, compatible with various VLMs