Scalable Visual Pretraining for Language Intelligence

2026-07-10

AI Summary

Traditional language model pretraining relies exclusively on plain text, disregarding the rich semantic visual information embedded in documents—such as figures, mathematical expressions, and layout—which results in incomplete knowledge representation. This work proposes an unsupervised visual pretraining paradigm that takes raw visual documents directly as input, bypassing intermediate text extraction and enabling end-to-end language intelligence learning. By challenging the prevailing “text-only” pretraining assumption, the approach demonstrates consistent performance gains across multiple benchmarks when applied to diverse backbone architectures, even under identical corpora. These results substantiate the potential of visual pretraining as an effective and scalable pathway toward enhanced language understanding.

Abstract

The rapid progress of large foundation models has been driven predominantly by pretraining on large-scale text corpora. However, many forms of knowledge are conveyed through visual representations, where figures, typeset equations, and page layouts carry rich information that cannot be faithfully or completely captured by text alone. Yet current pretraining approaches discard these visual cues by converting visually rich sources, such as documents and web pages, into plain text for learning language intelligence. This paper challenges the default assumption that language models must be trained on text-only representations and shows that Visual Pretraining is a scalable learner for foundation model intelligence. To this end, we conduct a systematic study of unsupervised visual pretraining paradigms that directly leverage visual documents without text extraction. Across multiple backbones and benchmarks, visual pretraining on the same underlying corpora consistently outperforms text-only pretraining, offering an efficient pathway to scalable language intelligence.

Problem

• visual pretraining
• language intelligence
• foundation models
• visual documents
• text-only pretraining

Innovation

• Visual Pretraining
• Foundation Models
• Multimodal Learning
• Document Understanding
• Scalable Pretraining