Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning

2024-09-16

arXiv.org

AI Summary

Existing self-supervised learning (SSL) methods employ fixed-frequency frequency-domain masking and reconstruction losses, overlooking image-specific spectral responses and inducing distributional shifts between pre-trained representations and natural image distributions—leading to data-hungry fine-tuning, slow convergence, and poor generalization. To address this, we propose FOLK: the first SSL framework that adaptively selects Fourier mask frequencies based on each image’s spectral response. FOLK introduces a dual-branch knowledge distillation architecture—comprising original and filtered image branches—that jointly optimizes contrastive learning and frequency-domain compression, preserving semantic integrity while mitigating distribution shift. Evaluated on image classification, few-shot learning, and semantic segmentation, FOLK achieves state-of-the-art performance. Under identical data budgets, it enables faster fine-tuning convergence and significantly stronger generalization across diverse downstream tasks.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We present a novel frequency-based Self-Supervised Learning (SSL) approach that significantly enhances its efficacy for pre-training. Prior work in this direction masks out pre-defined frequencies in the input image and employs a reconstruction loss to pre-train the model. While achieving promising results, such an implementation has two fundamental limitations as identified in our paper. First, using pre-defined frequencies overlooks the variability of image frequency responses. Second, pre-trained with frequency-filtered images, the resulting model needs relatively more data to adapt to naturally looking images during fine-tuning. To address these drawbacks, we propose FOurier transform compression with seLf-Knowledge distillation (FOLK), integrating two dedicated ideas. First, inspired by image compression, we adaptively select the masked-out frequencies based on image frequency responses, creating more suitable SSL tasks for pre-training. Second, we employ a two-branch framework empowered by knowledge distillation, enabling the model to take both the filtered and original images as input, largely reducing the burden of downstream tasks. Our experimental results demonstrate the effectiveness of FOLK in achieving competitive performance to many state-of-the-art SSL methods across various downstream tasks, including image classification, few-shot learning, and semantic segmentation.

Problem

• Enhance SSL by addressing pre-defined frequency masking limitations.
• Adaptively select masked frequencies based on image responses.
• Reduce downstream task burden with knowledge distillation framework.

Innovation

• Adaptive frequency masking based on image responses
• Two-branch framework with knowledge distillation
• Integration of filtered and original image inputs