Texture Representations in Deep Vision Models: Comparing CNNs, Vision Transformers, and Human Perception

2026-07-09

AI Summary

This study investigates the alignment between deep visual models and the human visual system in texture perception, moving beyond the conventional object-centric paradigm. By constructing stimulus sets of varying complexity using three source-image-based texture synthesis algorithms and leveraging human psychophysical data, the authors systematically compare the internal representational properties of convolutional neural networks (CNNs) and Vision Transformers (ViTs). The findings reveal that ViTs exhibit highly consistent texture representations that are insensitive to stimulus complexity and significantly better predict human texture discrimination behavior than CNNs. These results underscore network architecture as a critical determinant of texture representation, suggesting that ViTs more closely approximate human visual mechanisms in processing texture information.

Abstract

In computational vision science, Convolutional Neural Networks (CNNs) have emerged as a popular model of biological vision because of the alignment they can exhibit with neural and behavioral data in humans and animals. However, it remains unclear to what extent this alignment persists for visual tasks that extend beyond the canonical object recognition paradigm based on well defined semantic content. In this study, we diverge from the common object-centric view by focusing on another aspect of vision: texture perception. We consider textures of different complexity generated with three different algorithms from the same source images. Using a rank-based statistic, we quantify the information encoded in the internal representations of a CNN and three Vision Transformers (ViTs), and we compare the similarity of these representations to those inferred from human psychophysics data. We find that the representation of textures is aligned in different ViTs, but not between the ViTs and the CNN; that ViTs form similar representations for textures of different complexity; that human performance in recognizing textures can be better predicted from ViTs representations rather than CNN representations. Taken together, these results suggest that ViTs may capture more faithfully than CNNs how texture patterns are visually processed by humans, and that the representations of texture stimuli in computational models may be driven by the network architecture.

Problem

• texture perception
• deep vision models
• human perception
• representation alignment
• visual tasks

Innovation

• texture perception
• Vision Transformers
• Convolutional Neural Networks
• human psychophysics
• representation similarity