Integrating Language-Image Prior into EEG Decoding for Cross-Task Zero-Calibration RSVP-BCI

2025-01-06

AI Summary

RSVP-BCI systems suffer from poor cross-task generalization due to the need for task-specific calibration data, hindering practical deployment. Method: We introduce ELIPformer, a novel zero-calibration decoding framework leveraging semantic priors from vision-language pretraining. To support this, we construct the first open-source multi-task EEG–image paired dataset. ELIPformer incorporates a CLIP-driven prompt encoder that jointly models EEG temporal dynamics and establishes cross-modal bidirectional attention between neural signals and image semantics. Contribution/Results: ELIPformer achieves significant improvements in cross-task target identification accuracy without any task-specific calibration data—demonstrating the first semantic-guided zero-calibration RSVP decoding. This advances RSVP-BCI toward real-world applicability across diverse scenarios by eliminating the need for per-task recalibration.

Technology Category

• Computer Vision: Multi-modal Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Rapid Serial Visual Presentation (RSVP)-based Brain-Computer Interface (BCI) is an effective technology used for information detection by detecting Event-Related Potentials (ERPs). The current RSVP decoding methods can perform well in decoding EEG signals within a single RSVP task, but their decoding performance significantly decreases when directly applied to different RSVP tasks without calibration data from the new tasks. This limits the rapid and efficient deployment of RSVP-BCI systems for detecting different categories of targets in various scenarios. To overcome this limitation, this study aims to enhance the cross-task zero-calibration RSVP decoding performance. First, we design three distinct RSVP tasks for target image retrieval and build an open-source dataset containing EEG signals and corresponding stimulus images. Then we propose an EEG with Language-Image Prior fusion Transformer (ELIPformer) for cross-task zero-calibration RSVP decoding. Specifically, we propose a prompt encoder based on the language-image pre-trained model to extract language-image features from task-specific prompts and stimulus images as prior knowledge for enhancing EEG decoding. A cross bidirectional attention mechanism is also adopted to facilitate the effective feature fusion and alignment between the EEG and language-image features. Extensive experiments demonstrate that the proposed model achieves superior performance in cross-task zero-calibration RSVP decoding, which promotes the RSVP-BCI system from research to practical application.

Problem

• RSVP-BCI
• Task Adaptability
• Multi-object Recognition Efficiency

Innovation

• ELIPformer
• Attention Mechanism
• Integrated Pre-knowledge