Better Language Models Exhibit Higher Visual Alignment

2024-10-09

AI Summary

This work investigates whether purely text-based large language models (LLMs) inherently possess visual alignment capabilities. Method: We propose a zero-shot generalization framework that freezes the text encoder and integrates a discriminative vision–language architecture—requiring neither fine-tuning nor paired vision–language data. Contribution/Results: We empirically demonstrate, for the first time, that decoder-only LLMs intrinsically encode strong visual-semantic alignment, with this capability scaling positively with model size. Moreover, the approach supports cross-lingual zero-shot transfer. On Chinese cross-lingual zero-shot image classification, it achieves 38.7% accuracy—outperforming CLIP by 1.4 percentage points. Critically, it substantially reduces dependence on labeled data and computational resources while enhancing model robustness, generalization, and accessibility.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

How well do text-only Large Language Models (LLMs) naturally align with the visual world? We provide the first direct analysis by utilizing frozen text representations in a discriminative vision-language model framework and measuring zero-shot generalization on unseen classes. We find decoder-based LLMs exhibit high intrinsic visual alignment. In particular, more capable LLMs reliably demonstrate stronger generalization. Moreover, utilizing frozen LLMs leads to strong gains in cross-lingual settings, where our approach surpasses CLIP's accuracy of 1.4% with 38.7% for Chinese. Our proposed method improves both robustness and generalization and also significantly reduces the need for paired data and compute, making vision-language models more accessible and adaptable.

Problem

• Assessing text-only LLMs' visual alignment
• Evaluating zero-shot generalization in vision-language models
• Enhancing cross-lingual accuracy and reducing data needs

Innovation

• Utilizes frozen text representations
• Measures zero-shot generalization
• Reduces need for paired data