DS-MTNet:Structured Multi-Task EEG Decoding for Human-Machine Collaboration

2026-07-07

AI Summary

Existing approaches struggle to simultaneously decode multiple cognitive processes—such as event perception, response preparation, and vigilance—from continuous electroencephalography (EEG) signals in human–machine collaboration scenarios. This work proposes DS-MTNet, a novel framework that, for the first time, unifies three types of EEG evidence into a structured slot-filling representation. It integrates EEG waveforms, time–frequency power, and task-routing source embeddings through a multi-stream neural architecture, enhanced by a dual-gating mechanism to enable joint multi-task decoding. Evaluated on a sustained-attention driving dataset, DS-MTNet significantly outperforms both single-task and multi-task baselines, with the most pronounced gains observed during steering-response phases. The method achieves reusable, high-precision structured decoding of brain activity, advancing the feasibility of real-time cognitive state inference in collaborative settings.

Abstract

Current human-machine collaboration (HMC) systems rely on environment-facing sensors to observe visible actions and scene states, but the internal perceptual, intention-related, and state-related processes of operators remain insufficiently integrated into machine perception. Electroencephalography (EEG) provides a non-invasive, time-resolved modality to capture neural activity associated with these processes and can serve as an additional sensing channel in HMC. However, HMC-relevant EEG evidence is often mixed in continuous recordings. Existing EEG decoding methods usually target task-specific classification or aggregate prediction, so multiple HMC-relevant readouts are rarely organized in a unified EEG representation. To address this gap, this paper proposed the Decomposed-Source Multi-Task Network (DS-MTNet), a structured multi-task EEG decoding framework. DS-MTNet integrated three streams, namely EEG waveforms, task-routed source embeddings, and temporal-spectral power features, into reusable slots and used dual gating mechanisms to route task-specific components. The model was tested on a sustained-attention driving EEG dataset with three representative readouts: lane-departure-related epochs for environmental-event processing, steering-response stage for response preparation, and reaction-time-defined alertness state for internal state. DS-MTNet achieved the best mean performance among traditional, single-task deep, and multi-task EEG baselines, with the most robust gains observed for steering-response stage decoding. Ablation and interpretability analyses suggested that DS-MTNet jointly decoded multiple readouts and organized event-related, response-related, and state-related EEG evidence in a unified source-slot representation. These findings provide a computational step toward incorporating operator-related neural evidence into machine perception in HMC.

Problem

• human-machine collaboration
• EEG decoding
• multi-task learning
• neural evidence integration
• operator state perception

Innovation

• multi-task EEG decoding
• structured neural representation
• dual gating mechanism
• source-slot architecture
• human-machine collaboration