🤖 AI Summary
This study systematically investigates the transferability of Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) as backbones for few-shot geometric estimation—specifically 2D rigid transformation estimation and fundamental matrix prediction. Motivated by the unclear generalization capabilities of pretrained models in low-data regimes (e.g., autonomous driving, robotics), we fine-tune diverse pretrained models—including ResNet, EfficientNet, CLIP-ResNet, CLIP-ViT, and DINO—within a unified framework. Results show that CNNs, benefiting from strong inductive biases, achieve stable performance and superior cross-domain generalization under extreme data scarcity (≤1k samples), whereas ViTs significantly outperform CNNs with larger training sets and demonstrate greater out-of-distribution robustness. Our key contribution is the first empirical revelation of a strong coupling between architectural choice and data scale in geometric vision tasks, leading to the practical principle of “backbone selection conditioned on data scale.” This work establishes a reproducible benchmark and design paradigm for resource-constrained geometric learning.
📝 Abstract
Vision-transformers (ViTs) and large-scale convolution-neural-networks (CNNs) have reshaped computer vision through pretrained feature representations that enable strong transfer learning for diverse tasks. However, their efficiency as backbone architectures for geometric estimation tasks involving image deformations in low-data regimes remains an open question. This work considers two such tasks: 1) estimating 2D rigid transformations between pairs of images and 2) predicting the fundamental matrix for stereo image pairs, an important problem in various applications, such as autonomous mobility, robotics, and 3D scene reconstruction. Addressing this intriguing question, this work systematically compares large-scale CNNs (ResNet, EfficientNet, CLIP-ResNet) with ViT-based foundation models (CLIP-ViT variants and DINO) in various data size settings, including few-shot scenarios. These pretrained models are optimized for classification or contrastive learning, encouraging them to focus mostly on high-level semantics. The considered tasks require balancing local and global features differently, challenging the straightforward adoption of these models as the backbone. Empirical comparative analysis shows that, similar to training from scratch, ViTs outperform CNNs during refinement in large downstream-data scenarios. However, in small data scenarios, the inductive bias and smaller capacity of CNNs improve their performance, allowing them to match that of a ViT. Moreover, ViTs exhibit stronger generalization in cross-domain evaluation where the data distribution changes. These results emphasize the importance of carefully selecting model architectures for refinement, motivating future research towards hybrid architectures that balance local and global representations.