From Pixels to Components: Eigenvector Masking for Visual Representation Learning

2025-02-10

AI Summary

Existing self-supervised image learning methods employing random pixel-wise masking suffer from local redundancy in reconstruction, hindering effective high-level semantic representation learning. To address this, we propose a variance-aware, component-level masking strategy operating in the PCA feature space: principal components are stochastically masked according to their cumulative variance contribution, and representation learning is driven by a reconstruction objective in this transformed domain. This work constitutes the first approach to shift masking from the entangled pixel space to a disentangled, semantics-oriented component space—thereby compelling models to rely on global structural cues rather than local textures for inference. Evaluated on multiple downstream image classification benchmarks, the learned representations consistently outperform pixel-based baselines—including Masked Autoencoders—demonstrating superior discriminability and generalization capability.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Representation Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web

Abstract

Predicting masked from visible parts of an image is a powerful self-supervised approach for visual representation learning. However, the common practice of masking random patches of pixels exhibits certain failure modes, which can prevent learning meaningful high-level features, as required for downstream tasks. We propose an alternative masking strategy that operates on a suitable transformation of the data rather than on the raw pixels. Specifically, we perform principal component analysis and then randomly mask a subset of components, which accounts for a fixed ratio of the data variance. The learning task then amounts to reconstructing the masked components from the visible ones. Compared to local patches of pixels, the principal components of images carry more global information. We thus posit that predicting masked from visible components involves more high-level features, allowing our masking strategy to extract more useful representations. This is corroborated by our empirical findings which demonstrate improved image classification performance for component over pixel masking. Our method thus constitutes a simple and robust data-driven alternative to traditional masked image modeling approaches.

Problem

• Eigenvector masking for visual representation learning
• Predicting masked components from visible ones
• Improving image classification performance

Innovation

• Eigenvector masking strategy
• Principal component analysis
• Global information extraction