Vision Transformers Learn Gestalt-Like Figure-Ground Cues from Natural Images

2026-07-09

AI Summary

This study investigates how visual systems spontaneously learn Gestalt figure-ground organization cues—such as surroundedness, convexity, and symmetry—from natural images. By fitting linear probes on Vision Transformer models trained across 25 diverse objectives (spanning supervised and self-supervised paradigms) and evaluating performance on both natural images and synthetic stimuli, the work provides the first systematic evidence that such models automatically acquire key shape-based cues relied upon by human vision through exposure to natural scene statistics. The results demonstrate that the models robustly encode surroundedness and convexity, exhibit sensitivity to symmetry in uniform-colored regions but limited efficacy in textured areas, and crucially, show zero-shot generalization from natural-image-trained probes to synthetic stimuli, indicating strong cross-stimulus-type generalization capabilities.

Abstract

Figure-ground organization in the human visual system relies on several shape-based cues, including surroundedness, convexity, and symmetry. While these cues have been extensively studied using abstract stimuli, little is known about how they operate under natural conditions or how they arise from the statistics of natural scenes. Deep neural networks offer a promising path forward: a model that relies on the same figure-ground cues as humans would provide tractable experimental access to the underlying mechanisms. In this study, we evaluate shape-based figure-ground organization in Vision Transformers (ViTs), for which prior work has demonstrated the emergence of object-based grouping. We test 25 ViTs spanning supervised and self-supervised training objectives, by fitting linear probes to predict figure-ground assignment from intermediate patch representations using both natural images and controlled artificial stimuli that isolate individual cues. Our results show that ViTs robustly encode surroundedness and convexity, and that probes trained on natural images generalize zero-shot to artificial stimuli across several models. For symmetry we observe mixed results: the cue is encoded for uniformly colored but not for textured regions. Taken together, our findings demonstrate that Gestalt-like figure-ground cues can be learned from natural scene statistics and position ViTs as a compelling model system for studying the computational mechanisms of perceptual organization. Code and data is available at https://github.com/mtangemann/mlvbench

Problem

• figure-ground organization
• Gestalt cues
• natural scene statistics
• shape-based cues
• Vision Transformers

Innovation

• Vision Transformers
• figure-ground organization
• Gestalt cues
• natural scene statistics
• linear probing