Vision-TTT: Efficient and Expressive Visual Representation Learning with Test-Time Training

2026-02-28

AI Summary

This work addresses the challenge of balancing efficiency and representational capacity in Vision Transformers, whose self-attention mechanism incurs quadratic computational complexity. The authors introduce ViTTT, a novel backbone architecture that, for the first time, adapts the linear-complexity Test-Time Training (TTT) framework to vision tasks. ViTTT employs self-supervised learning to compress visual token sequences and integrates a bidirectional scanning strategy with Conv2d modules to efficiently capture global image dependencies. On ImageNet, ViTTT achieves Top-1 accuracy ranging from 77.3% to 82.5%. Notably, ViTTT-T at 1280×1280 resolution reduces FLOPs by 79.4%, accelerates inference by 4.38×, and decreases memory usage by 88.9% compared to DeiT-T, while significantly outperforming existing efficient models on downstream tasks.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Learning efficient and expressive visual representation has long been the pursuit of computer vision research. While Vision Transformers (ViTs) gradually replace traditional Convolutional Neural Networks (CNNs) as more scalable vision learners, their applications are plagued by the quadratic complexity of the self-attention mechanism. To address the challenge, we introduce a new linear-time sequence modeling method Test-Time Training (TTT) into vision and propose Vision-TTT, which compresses the visual token sequence in a novel self-supervised learning manner. By incorporating bidirectional scan strategy and the Conv2d module, Vision-TTT effectively extends vanilla TTT to model 2D visual correlations with global receptive fields. Extensive experiments show that \texttt{Vittt-T/S/B} achieve 77.3%,81.2%,82.5% Top-1 accuracy on ImageNet classification and also greatly outperform their counterparts on downstream tasks. At 1280x1280 resolution, \texttt{Vittt-T} reduces FLOPs by 79.4% and runs 4.38x faster with 88.9% less memory than DeiT-T. These results demonstrate the expressiveness and efficiency of Vision-TTT as a strong candidate for the next-generation generic visual backbone.

Problem

• visual representation learning
• Vision Transformers
• self-attention complexity
• efficient vision models
• quadratic complexity

Innovation

• Vision-TTT
• Test-Time Training
• linear-time attention
• visual representation learning
• efficient vision models