iGVLM: Dynamic Instruction-Guided Vision Encoding for Question-Aware Multimodal Understanding

2026-03-03

AI Summary

Current large vision-language models struggle with fine-grained reasoning due to their reliance on static, instruction-agnostic visual encoders that cannot dynamically attend to task-relevant visual cues. To address this limitation, this work proposes iGVLM, a novel framework featuring instruction-guided dynamic visual encoding. iGVLM employs a decoupled dual-branch architecture: one branch remains frozen to preserve pretrained visual priors, while the other leverages adaptive layer normalization (AdaLN) to modulate visual features conditioned on input instructions, enabling a smooth transition from generic perception to instruction-aware reasoning. The method substantially enhances instruction sensitivity and logical consistency across multiple benchmarks, demonstrates compatibility with diverse language backbones, and is validated on a newly introduced MM4 diagnostic dataset, showcasing superior performance in multi-query and multi-instruction scenarios.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Despite the success of Large Vision--Language Models (LVLMs), most existing architectures suffer from a representation bottleneck: they rely on static, instruction-agnostic vision encoders whose visual representations are utilized in an invariant manner across different textual tasks. This rigidity hinders fine-grained reasoning where task-specific visual cues are critical. To address this issue, we propose iGVLM, a general framework for instruction-guided visual modulation. iGVLM introduces a decoupled dual-branch architecture: a frozen representation branch that preserves task-agnostic visual representations learned during pre-training, and a dynamic conditioning branch that performs affine feature modulation via Adaptive Layer Normalization (AdaLN). This design enables a smooth transition from general-purpose perception to instruction-aware reasoning while maintaining the structural integrity and stability of pre-trained visual priors. Beyond standard benchmarks, we introduce MM4, a controlled diagnostic probe for quantifying logical consistency under multi-query, multi-instruction settings. Extensive results show that iGVLM consistently enhances instruction sensitivity across diverse language backbones, offering a plug-and-play paradigm for bridging passive perception and active reasoning.

Problem

• Large Vision-Language Models
• representation bottleneck
• instruction-agnostic vision encoding
• task-specific visual cues
• multimodal understanding

Innovation

• instruction-guided vision encoding
• dynamic feature modulation
• Adaptive Layer Normalization (AdaLN)
• decoupled dual-branch architecture
• multimodal reasoning