EEG Benchmarking Needs a Task Specification Layer: NeuroDoc for Rulebook-Guided, Executable Benchmark Construction

2026-06-22

AI Summary

This study addresses the lack of standardized task specifications in existing public electroencephalography (EEG) datasets, which hinders the conversion of heterogeneous data into reusable and auditable benchmark units. To resolve this, the authors propose a framework that integrates a structured task specification language with a shared rulebook, aligning task documentation with executable task kernels to enable standardized EEG task definitions. The work introduces two novel mechanisms—NeuroDoc for rule-guided documentation drafting and NeuroAudit for community review and version control—thereby establishing the first executable and auditable EEG benchmarking system. The released benchmark corpus comprises 53 review criteria and 245 task definitions, demonstrating strong reusability and executability across four foundational EEG models.

Abstract

Electroencephalography (EEG) foundation models increasingly rely on multi-dataset training and evaluation, yet public EEG datasets still lack a shared task specification layer that can turn heterogeneous recordings into reusable benchmark units. Existing standards organize files, metadata, and provenance, but they do not specify EEG tasks under a common language and rulebook, leaving critical task semantics scattered across papers, code, and manual interpretation. We investigate whether heterogeneous public EEG datasets can be standardized through a structured task specification language paired with a shared rulebook. Our methodology represents each benchmark entry as a task document synchronized with an executable task kernel, with the rulebook defining task fields, evidence requirements, document-kernel alignment, review states, and machine-checkable constraints. Using this methodology, we release a community-reviewed EEG benchmark corpus centered on 53 completed and reviewed entries with 245 task definitions spanning diverse paradigms, and we introduce NeuroDoc and NeuroAudit as the operational support layer for rulebook-guided drafting, upgrading, review, amendment, and release management. We further examine whether the resulting benchmark units can be instantiated in a shared downstream setting across four EEG foundation model backbones, providing execution-based evidence for reusable, auditable, and executable EEG benchmarking infrastructure.

Problem

• EEG benchmarking
• task specification
• foundation models
• standardization
• reproducibility

Innovation

• task specification
• executable benchmark
• EEG foundation models
• NeuroDoc
• rulebook-guided standardization