Self-Evolving Visual Concept Library using Vision-Language Critics

2025-03-31

AI Summary

This work addresses the challenge of unsupervised construction of highly discriminative concept libraries for visual recognition. Methodologically, it introduces the first fully automated, annotation-free, self-evolving framework that integrates vision-language models (VLMs) as critics for concept evaluation and filtering, leverages large language models (LLMs) with in-context learning to dynamically refine concept generation strategies, and explicitly models inter-concept relationships. Through iterative cycles of discovery, assessment, and reconstruction—performed entirely without supervision—the framework enables autonomous concept library evolution. Extensive experiments demonstrate significant performance gains across zero-shot, few-shot, and fine-tuned visual classification benchmarks, validating the framework’s effectiveness, generalizability, and plug-and-play practicality.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We study the problem of building a visual concept library for visual recognition. Building effective visual concept libraries is challenging, as manual definition is labor-intensive, while relying solely on LLMs for concept generation can result in concepts that lack discriminative power or fail to account for the complex interactions between them. Our approach, ESCHER, takes a library learning perspective to iteratively discover and improve visual concepts. ESCHER uses a vision-language model (VLM) as a critic to iteratively refine the concept library, including accounting for interactions between concepts and how they affect downstream classifiers. By leveraging the in-context learning abilities of LLMs and the history of performance using various concepts, ESCHER dynamically improves its concept generation strategy based on the VLM critic's feedback. Finally, ESCHER does not require any human annotations, and is thus an automated plug-and-play framework. We empirically demonstrate the ability of ESCHER to learn a concept library for zero-shot, few-shot, and fine-tuning visual classification tasks. This work represents, to our knowledge, the first application of concept library learning to real-world visual tasks.

Problem

• Automating visual concept library creation without human annotation
• Improving concept discriminative power via iterative VLM feedback
• Enhancing zero-shot and few-shot visual classification tasks

Innovation

• Iterative concept refinement with vision-language critics
• Dynamic concept generation using VLM feedback
• Automated plug-and-play framework without human annotations