Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization

2024-12-24

AI Summary

Zero-shot generalization in computer vision (CV) lags behind natural language processing (NLP), primarily because conventional CV task definitions (e.g., “image segmentation”) are highly specialized and discrete, hindering cross-task transfer. Method: This work proposes explicitly specifying input-to-output mappings via natural language instructions, enabling task-agnostic formulation. We construct a large-scale multimodal instruction dataset comprising 12 million image–instruction–output triplets and design an autoregressive vision-language model that jointly optimizes image-text embedding and generative modeling. Contribution/Results: Our key innovation lies in shifting task representation from discrete labels to interpretable, composable natural language instructions—enabling unified, cross-task semantic grounding. Experiments demonstrate strong instruction-level zero-shot performance on seen tasks and substantial gains over prior methods on unseen CV tasks. The framework establishes a unified, interpretable, and generalizable paradigm for visual task understanding.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Computer Vision (CV) has yet to fully achieve the zero-shot task generalization observed in Natural Language Processing (NLP), despite following many of the milestones established in NLP, such as large transformer models, extensive pre-training, and the auto-regression paradigm, among others. In this paper, we explore the idea that CV adopts discrete and terminological task definitions (eg, ``image segmentation''), which may be a key barrier to zero-shot task generalization. Our hypothesis is that without truly understanding previously-seen tasks--due to these terminological definitions--deep models struggle to generalize to novel tasks. To verify this, we introduce Explanatory Instructions, which provide an intuitive way to define CV task objectives through detailed linguistic transformations from input images to outputs. We create a large-scale dataset comprising 12 million ``image input $ o$ explanatory instruction $ o$ output'' triplets, and train an auto-regressive-based vision-language model (AR-based VLM) that takes both images and explanatory instructions as input. By learning to follow these instructions, the AR-based VLM achieves instruction-level zero-shot capabilities for previously-seen tasks and demonstrates strong zero-shot generalization for unseen CV tasks. Code and dataset will be openly available on our GitHub repository.

Problem

• Zero-shot Learning
• Computer Vision
• Natural Language Processing

Innovation

• Explainable Guidance
• Zero-Shot Learning
• Computer Vision