Show Me Examples: Inferring Visual Concepts from Image Sets

2026-07-02

AI Summary

Existing vision-language models struggle to infer shared abstract visual concepts from a small set of example images and generalize them to novel images. This work introduces the Visual Concept Induction and Synthesis (VICIS) task—the first formalization of learning and transferring visual concepts purely from visual exemplars. We propose an end-to-end trainable architecture that integrates image set encoding, concept embedding extraction, and conditional image generation, enabling generalization across modalities (e.g., sketches) and to unseen concepts. Experiments on both synthetic data and large-scale benchmarks derived from ImageNet and WordNet demonstrate that our method significantly outperforms existing approaches in terms of both accuracy and diversity of generated images.

Abstract

Vision-language models (VLMs) can follow complex textual instructions, yet they struggle to reason from purely visual context. In particular, current models fail to infer shared concepts from sets of example images and apply them to new inputs. We introduce Visual Concept Inference from Sets (VICIS), a task that evaluates this capability. Given a small context set of images sharing a concept and a query image, the model must generate new images that preserve the context-defined concept while remaining consistent with the query. We show that state-of-the-art VLMs perform poorly on this task, often ignoring the visual context or defaulting to biased generations. To address this gap, we propose a training framework and architecture that learn to infer visual concepts from image sets and extract concept-specific embeddings from queries. Experiments on synthetic data and large-scale ImageNet/WordNet data show that our model generates more accurate and diverse outputs and generalizes to unseen concepts and modalities such as sketches.

Problem

• visual concept inference
• vision-language models
• image sets
• concept generalization
• visual reasoning

Innovation

• visual concept inference
• image set reasoning
• vision-language models
• concept-specific embedding
• cross-modal generalization