RATS! Patches Talk Through Registers: Emergent Parts in Register Attention Transformers

2026-06-12

AI Summary

This work addresses the challenge of enabling self-supervised vision models to autonomously discover reusable, structured object part representations without any annotations. To this end, the authors propose the RATS architecture, which decomposes the classification token into multiple learnable register tokens and introduces a compress–communicate–broadcast three-stage attention mechanism (L→N→N→L). This design facilitates spontaneous disentanglement and identification of object parts without auxiliary losses or part-level supervision. Notably, RATS achieves cross-category semantically consistent and stable part representations under a purely self-supervised setting for the first time, demonstrating strong generalization: it improves performance by an average of 12 mIoU across five segmentation benchmarks, with gains of +1.11 mIoU on ADE20K and +0.2 AP^m on COCO, validating the efficacy and versatility of the register-based mechanism.

Abstract

When humans see a bird, they recognize far more than just "bird" -- they see a head, wings, and talons, a structured assembly of reusable parts that can be identified across every bird they have ever seen. We ask whether a self-supervised visual model can discover the same compositional structure on its own. To this end, we propose RATS (Register Attention Transformers), which decomposes the classification token into N learnable register tokens that route patch information through an L->N->N->L bottleneck via a three-step compress-communicate-broadcast attention. The N registers are partitioned across the H attention heads, so that registers assigned to different heads do not interact with each other. Without auxiliary losses or part annotations, each register spontaneously specializes into a proto-semantic region whose emerging structure resembles object parts. RATS surpasses all baselines by +12 mIoU on average across five segmentation benchmarks, with consistent gains on ADE20K (+1.11 mIoU) and COCO (+0.2 AP^m). Its register dictionary further exhibits part-level consistency and semantic proximity across related categories. Our results suggest that RATS may provide a useful architectural prior for structured and interpretable visual representation learning.

Problem

• self-supervised learning
• compositional structure
• object parts
• visual representation
• structured representation

Innovation

• Register Attention
• Emergent Parts
• Self-supervised Representation
• Structured Vision
• Transformer Bottleneck