Self-supervised pretraining for an iterative image size agnostic vision transformer

2026-04-22

AI Summary

This work addresses the computational inefficiency and limited adaptability to arbitrary image resolutions in existing vision Transformers during self-supervised pretraining. The authors propose a sequence-to-global self-supervised pretraining framework that, for the first time, integrates self-supervised learning into an iterative multi-scale Vision Transformer architecture, enabling truly resolution-agnostic pretraining. Building upon the DINO self-distillation objective, the method leverages integral-image-accelerated multi-scale patch extraction and a recursive, time-unrolled backpropagation mechanism to support inputs of any resolution while maintaining constant computational overhead. Experiments demonstrate competitive performance on ImageNet-1K and multiple downstream classification tasks, with inference computational cost remaining invariant to input resolution.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Unsupervised & Self-Supervised Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Web Mining and Content Analysis: Large pretrained models with web data
• Security and Privacy: Data transparency and provenance

Abstract

Vision Transformers (ViTs) dominate self-supervised learning (SSL). While they have proven highly effective for large-scale pretraining, they are computationally inefficient and scale poorly with image size. Consequently, foundational models like DINO are constrained to low-resolution processing. A recent foveal-inspired transformer achieves resolution agnosticism by iteratively processing a fixed-size context of multi-zoom patches. This model demonstrated promising results via supervised learning, utilizing a sequential, recurrent-like process without backpropagation through time. To unlock its potential as a foundational backbone, we introduce a novel sequential-to-global SSL framework based on DINO's self-distillation objective. Supported by an efficient integral-image patch extraction method, our approach enables large-scale pretraining for image-size agnostic vision encoders. We achieve competitive performance on ImageNet-1K and downstream classification tasks, maintaining a constant computational budget regardless of input resolution.

Problem

• Vision Transformers
• self-supervised learning
• image size agnostic
• computational efficiency
• resolution scalability

Innovation

• self-supervised learning
• vision transformer
• resolution agnostic
• iterative processing
• integral-image patch extraction