CATVis: Context-Aware Thought Visualization

2025-07-15

AI Summary

This study addresses the challenge of high-fidelity visual representation decoding and image generation from noisy, low signal-to-noise ratio (SNR) electroencephalography (EEG) signals. We propose a five-stage context-aware EEG-to-Image generation framework: (1) time-frequency feature extraction via a dedicated EEG encoder; (2) cross-modal alignment into the CLIP visual semantic space; (3) caption re-ranking and weighted interpolation to enhance semantic consistency; (4) diffusion-based image synthesis driven by aligned embeddings; and (5) end-to-end optimization integrating concept classification, caption reconstruction, and diffusion priors. Our key contribution is the joint optimization of these objectives to achieve direct EEG-to-image alignment. Experiments demonstrate significant improvements over state-of-the-art EEG decoding methods: +13.43% in classification accuracy, +15.21% in generation accuracy, and −36.61% in Fréchet Inception Distance (FID).

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

EEG-based brain-computer interfaces (BCIs) have shown promise in various applications, such as motor imagery and cognitive state monitoring. However, decoding visual representations from EEG signals remains a significant challenge due to their complex and noisy nature. We thus propose a novel 5-stage framework for decoding visual representations from EEG signals: (1) an EEG encoder for concept classification, (2) cross-modal alignment of EEG and text embeddings in CLIP feature space, (3) caption refinement via re-ranking, (4) weighted interpolation of concept and caption embeddings for richer semantics, and (5) image generation using a pre-trained Stable Diffusion model. We enable context-aware EEG-to-image generation through cross-modal alignment and re-ranking. Experimental results demonstrate that our method generates high-quality images aligned with visual stimuli, outperforming SOTA approaches by 13.43% in Classification Accuracy, 15.21% in Generation Accuracy and reducing Fréchet Inception Distance by 36.61%, indicating superior semantic alignment and image quality.

Problem

• Decoding visual representations from noisy EEG signals
• Improving EEG-to-image generation with semantic alignment
• Enhancing image quality and accuracy in BCIs

Innovation

• EEG encoder for concept classification
• Cross-modal alignment in CLIP space
• Stable Diffusion for image generation