TuringViT: Making SOTA Vision Transformers Accessible to All

2026-06-23

AI Summary

This work proposes an efficient and scalable Vision Transformer (ViT) training framework that addresses the high computational cost of standard attention mechanisms in high- or dynamic-resolution settings and the heavy reliance on massive image-text datasets. The approach introduces three key innovations: the first-of-its-kind Turing Linear Attention (TLA) mechanism, a high-quality image-text-video dataset curated via the VISTA-Curation strategy, and a pretraining paradigm natively supporting dynamic input resolutions. Remarkably, the model achieves superior performance using only 10% of the data required by leading open-source ViTs, demonstrates enhanced accuracy on downstream vision-language tasks, significantly reduces inference latency at high resolutions, and exhibits strong data scalability and deployment flexibility.

Abstract

Modern VLMs and VLA systems commonly adopt off-the-shelf ViTs such as SigLIP2 as visual encoders, but diverse downstream requirements in latency, temporal modeling, and VLM integration often call for customized SOTA-level ViTs. Training such encoders remains beyond the reach of much of the community, as it requires massive image-text data, while standard softmax attention makes high-resolution or dynamic-resolution pretraining prohibitively costly and often forces low-resolution pretraining followed by post-hoc adaptation. TuringViT addresses these challenges with three key designs: Turing Linear Attention (TLA) for efficient sequence modeling, VISTA-Curation to construct supervision-rich image-video training data, and native dynamic-resolution pretraining that supports flexible inputs from the start and transfers seamlessly to downstream VLMs. As a result, TuringViT outperforms leading open-source ViT baselines with only 10% of the data, achieves stronger downstream VLM performance, and delivers substantially better latency scaling on high-resolution inputs. Our scaling-law analysis further shows that TuringViT continues to improve predictably with curated data scale, far from saturation. Its fast adaptation, hardware-friendly design, and efficient deployment have made it a unified visual foundation across XPeng's AI systems. More broadly, TuringViT provides a reproducible pipeline that dramatically lowers the cost for the community to train, customize, and deploy SOTA-level ViTs, moving toward making such Vision Transformers accessible to all.

Problem

• Vision Transformers
• high-resolution pretraining
• dynamic-resolution
• computational cost
• customization

Innovation

• Turing Linear Attention
• dynamic-resolution pretraining
• VISTA-Curation
• vision transformers
• efficient sequence modeling