LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment

2026-05-04

AI Summary

This work addresses the instability and error propagation in fine-grained zero-shot recognition caused by existing methods’ reliance on excessive redundant region proposals or premature semantic guidance. The authors propose an efficient and robust local visual–textual alignment framework that first obtains stable visual initialization through class-agnostic object-centric region discovery, then adaptively applies language guidance based on intermediate prediction confidence for refinement, and finally integrates evidence from object, context, and global representations. Key innovations include a class-agnostic region discovery mechanism, a confidence-adaptive language guidance strategy, and a dual-path object–context aggregation module. The method achieves state-of-the-art performance on both standard and distribution-shift benchmarks while significantly reducing the number of required region proposals.

Abstract

Zero-shot recognition aims to classify an image by selecting the most compatible label description from a set of candidate classes without any task-specific supervision. In fine-grained settings, however, the relevant evidence often lies in localized parts, attributes, or textures rather than in the full image, making whole-image alignment suboptimal. Recent localized visual-text alignment methods address this by comparing class descriptions with multiple image regions, but they typically rely on large sets of random or redundant crops, increasing inference cost and introducing many highly redundant or weakly relevant candidates. Moreover, introducing semantic guidance too early can create an error-amplifying feedback process in which inaccurate intermediate predictions bias later localization and reinforce subsequent mistakes; we refer to this failure mode as the prediction loop. We propose LAGO (LAnguage-Guided adaptive Object-region focus), a framework for efficient and robust zero-shot localized visual-text alignment. LAGO first performs class-agnostic object-centric candidate discovery to obtain a stable visual initialization, and then applies adaptive language-guided refinement with the strength of semantic guidance controlled by intermediate confidence. It further combines object-level, contextual, and full-image evidence through an effective object-context dual-channel aggregation strategy. Extensive experiments show that LAGO consistently achieves state-of-the-art performance on standard zero-shot benchmarks and challenging distribution-shift settings, while requiring substantially fewer candidate regions at inference time.

Problem

• zero-shot recognition
• fine-grained alignment
• localized visual-text alignment
• prediction loop
• redundant region proposals

Innovation

• zero-shot recognition
• localized visual-text alignment
• language-guided refinement
• object-centric discovery
• dual-channel aggregation