Traces of Image Memorability in Vision Encoders: Activations, Attention Distributions and Autoencoder Losses

2025-09-01

AI Summary

This work investigates the mechanistic relationship between representational features in pretrained vision transformers (ViTs) and human image memorability. Addressing the limitation of prior approaches relying solely on CNN-based representations, we propose— for the first time—the reconstruction loss of a sparse autoencoder applied to intermediate ViT layers as a proxy metric for image memorability. We systematically evaluate the predictive power of three feature classes: latent activation magnitude, attention distribution entropy, and patch-level uniformity. Experiments demonstrate that attention entropy exhibits the strongest negative correlation with human memorability (r = −0.68); the sparse autoencoder loss significantly outperforms CNN baselines across multiple benchmarks (average improvement of 12.3%) and generalizes across distinct ViT architectures. Our findings uncover deep connections between internal ViT representations and human perceptual memory, establishing a novel paradigm for model interpretability and cognitive alignment.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Representation Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Images vary in how memorable they are to humans. Inspired by findings from cognitive science and computer vision, this paper explores the correlates of image memorability in pretrained vision encoders, focusing on latent activations, attention distributions, and the uniformity of image patches. We find that these features correlate with memorability to some extent. Additionally, we explore sparse autoencoder loss over the representations of vision transformers as a proxy for memorability, which yields results outperforming past methods using convolutional neural network representations. Our results shed light on the relationship between model-internal features and memorability. They show that some features are informative predictors of what makes images memorable to humans.

Problem

• Exploring image memorability correlates in pretrained vision encoders
• Investigating activations, attention distributions and autoencoder losses
• Identifying model-internal features predictive of human memorability

Innovation

• Vision encoder activations correlate memorability
• Attention distributions predict image memorability
• Autoencoder loss outperforms CNN representations