Language-Instructed Vision Embeddings for Controllable and Generalizable Perception

2026-06-17

AI Summary

This work addresses the limitations of traditional vision foundation models, which act as static feature extractors and often require large downstream models for task adaptation. The authors propose LIVE, a novel approach that leverages language instructions as high-level control signals to dynamically guide the visual encoder during inference, generating task-oriented embeddings without any retraining. LIVE introduces a language-guided visual embedding mechanism, instruction-driven context-aware encoding, and a unified vision-language reasoning architecture. Evaluated on the MMVP benchmark, the method significantly reduces visual hallucination (improving by 34 points), surpasses existing large vision-language models with substantially fewer parameters, and demonstrates strong generalization to unseen instructions and tasks.

Abstract

Vision foundation models are typically trained as static feature extractors, placing the burden of task adaptation onto large downstream models. We propose an alternative paradigm: instead of solely feeding visual features into language models, we use language itself to dynamically guide the vision encoder. Our method, Language-Instructed Vision Embeddings (LIVE), leverages language as high-level guidance to produce task-centric embeddings at inference time, removing the need for task-specific retraining. This enables the encoder to focus on contextually relevant aspects of the input, yielding more controllable and generalizable representations. Empirically, LIVE reduces visual hallucinations (+34 points on MMVP), surpasses vision-language models with orders of magnitude more parameters on visual question answering, and generalizes to unseen instructions and tasks -- offering a direct path toward adaptive, instruction-driven visual intelligence.

Problem

• vision foundation models
• task adaptation
• controllable perception
• generalizable representations
• language-guided vision

Innovation

• Language-Instructed Vision Embeddings
• vision foundation models
• instruction-driven perception
• controllable representation
• generalizable vision