DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models

2026-04-16

AI Summary

This work addresses the challenge of deploying EEG foundation models in embedded brain-computer interfaces, where existing knowledge distillation approaches underperform due to their neglect of intermediate-layer semantics and frequency-domain structure. To overcome these limitations, we propose DLink, a novel distillation framework that dynamically aggregates critical intermediate representations from the teacher model via adaptive routing, employs a “mimic-then-compress” student architecture to avoid heavy classification heads, and introduces spectral distillation to align frequency-domain representations and preserve intrinsic EEG oscillatory characteristics. By integrating dynamic layer aggregation, structured spatiotemporal compression, and frequency-aware alignment, DLink significantly outperforms current distillation baselines across four EEG benchmarks, achieving performance close to fully fine-tuned foundation models while substantially reducing model parameters and inference overhead.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Diffusion Models for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

EEG foundation models (FMs) achieve strong cross-subject and cross-task generalization but impose substantial computational and memory costs that hinder deployment on embedded BCI systems. Knowledge distillation is a natural solution; however, conventional methods fail for EEG FMs because task-relevant semantics are often distributed across intermediate layers, and aggressive dimensionality reduction can distort oscillatory structure via representational collapse and aliasing. To address these challenges, we propose DLink (Distilling Layer-wise and Dominant Knowledge), a unified framework for transferring knowledge from large EEG FMs to compact students with three key innovations: (1) a dynamic Router that adaptively aggregates teacher layers to capture dominant intermediate representations; (2) an EEG MiC student with a Mimic-then-Compress pipeline, which inherits high-dimensional teacher features and then applies structured spatio-temporal compression to avoid a heavy classification head; and (3) spectral distillation that aligns teacher-student representations in the frequency domain to regularize compression and mitigate aliasing and temporal jitter. Experiments on four EEG benchmarks show that DLink enables compact students to outperform lightweight baselines while approaching fully fine-tuned FM performance at substantially lower model size and inference cost.

Problem

• EEG foundation models
• knowledge distillation
• embedded BCI systems
• representational collapse
• spectral distortion

Innovation

• knowledge distillation
• EEG foundation models
• dynamic routing
• spectral distillation
• structured compression