Probability Distribution Alignment and Low-Rank Weight Decomposition for Source-Free Domain Adaptive Brain Decoding

2025-04-12

AI Summary

In passive domain-adaptive brain decoding, three key challenges arise: privacy leakage due to inter-subject variability, misalignment of marginal distributions across modalities, and prohibitive computational cost from high-dimensional CLIP embeddings. To address these, we propose: (1) the first joint alignment of softmax conditional distributions and marginal probability distributions; (2) an image–text–fMRI tri-modal co-alignment mechanism to explicitly model cross-modal interactions; and (3) a low-rank weight decomposition scheme to compress CLIP embeddings—reducing dimensionality while preserving representational fidelity. Critically, our method operates without any source-domain fMRI data, thereby ensuring subject privacy while maintaining strong generalization. Evaluated on multiple standard fMRI datasets, it achieves a 3.2 dB improvement in image reconstruction PSNR, an 18% reduction in LPIPS, and a 2.1× speedup in inference latency.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Multi-modal Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Brain decoding currently faces significant challenges in individual differences, modality alignment, and high-dimensional embeddings. To address individual differences, researchers often use source subject data, which leads to issues such as privacy leakage and heavy data storage burdens. In modality alignment, current works focus on aligning the softmax probability distribution but neglect the alignment of marginal probability distributions, resulting in modality misalignment. Additionally, images and text are aligned separately with fMRI without considering the complex interplay between images and text, leading to poor image reconstruction. Finally, the enormous dimensionality of CLIP embeddings causes significant computational costs. Although the dimensionality of CLIP embeddings can be reduced by ignoring the number of patches obtained from images and the number of tokens acquired from text, this comes at the cost of a significant drop in model performance, creating a dilemma. To overcome these limitations, we propose a source-free domain adaptation-based brain decoding framework

Problem

• Address individual differences without source data privacy issues
• Align marginal probability distributions for better modality alignment
• Reduce CLIP embedding dimensionality without performance loss

Innovation

• Probability distribution alignment for modality adaptation
• Low-rank weight decomposition to reduce dimensionality
• Source-free domain adaptation for privacy protection