Representation Learning with Adaptive Superpixel Coding

2025-08-21

AI Summary

Traditional vision Transformers suffer from fixed-grid patch partitioning, which hinders adaptability to diverse image content and limits semantic representation capability. To address this, we propose the Self-supervised Adaptive Superpixel Transformer (ASC), whose core innovation is a learnable, dynamic superpixel segmentation layer that replaces static patch embedding to enable content-aware, adaptive region aggregation. This layer is jointly optimized with the Transformer encoder in a fully self-supervised manner—requiring no human annotations—to learn structured visual representations. Evaluated on ImageNet classification as well as downstream detection and segmentation benchmarks, ASC consistently outperforms state-of-the-art methods including ViT and MAE. Results demonstrate superior generalization ability and enhanced semantic modeling capacity, validating the effectiveness of adaptive, content-driven spatial abstraction in vision Transformers.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Unsupervised & Self-Supervised Learning
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Deep learning vision models are typically tailored for specific modalities and often rely on domain-specific assumptions, such as the grid structures used by nearly all existing vision models. In this work, we propose a self-supervised model based on Transformers, which we call Adaptive Superpixel Coding (ASC). The key insight of our model is to overcome the limitations of traditional Vision Transformers, which depend on fixed-size and non-adaptive patch partitioning. Instead, ASC employs adaptive superpixel layers that dynamically adjust to the underlying image content. We analyze key properties of the approach that make it effective, and find that our method outperforms widely-used alternatives on standard image downstream task benchmarks.

Problem

• Overcoming fixed-size patch limitations in vision transformers
• Adapting image representation to dynamic content features
• Enhancing self-supervised learning for downstream vision tasks

Innovation

• Adaptive superpixel layers dynamically adjust content
• Self-supervised Transformer model overcomes fixed patch limitations
• Method outperforms alternatives on image benchmark tasks