Enhancing deep learning models for time series classification via knowledge distillation

2026-07-07

AI Summary

This study addresses the high computational and memory demands of deep learning models for time series classification, which hinder their deployment on resource-constrained devices. The authors systematically evaluate the effectiveness of knowledge distillation across three prominent architectures—FCN, Inception, and ConvTran—and demonstrate that student models of moderate complexity yield the most significant distillation gains. Specifically, an FCN student achieves a 38-fold reduction in parameter count, an Inception student matches teacher performance with 42% fewer parameters, and a dual-attention-head ConvTran student exhibits particularly notable improvements. The work provides a fully open-source implementation to facilitate reproducible research on efficient time series models.

Abstract

Deep learning has achieved remarkable success in various domains including time series analysis, computer vision and natural language processing. However, high computational and memory demands of state-of-the-art architectures pose challenges for deployment in resource-limited environments. Knowledge Distillation (KD) addresses this by transferring knowledge from a large teacher model to a smaller, more efficient student model while maintaining competitive performance. In this work, we investigate the effectiveness of KD for Time Series Classification (TSC) across three architectures: the classical Fully Convolutional Network (FCN), the convolutional Inception model and the transformer-based ConvTran model. We evaluate our approach on UCR Archive, the largest benchmark repository of time series datasets, by modifying architectural components such as convolutional filters, Inception modules and attention heads across the three architectures. Our results consistently show that KD most effectively benefits student models of intermediate complexity across all three architectures, with the distilled FCN student reducing parameters by a factor of 38, the distilled Inception student achieving nearly the same performance as the teacher with 42% fewer parameters and the distilled ConvTran student with 2 attention heads showing the most significant improvement through distillation. To encourage further research and reproducibility, we provide our implementation at https://github.com/MSD-IRIMAS/KD-4-TSC.

Problem

• Time Series Classification
• Knowledge Distillation
• Model Compression
• Resource-Limited Deployment
• Deep Learning

Innovation

• Knowledge Distillation
• Time Series Classification
• Model Compression
• Deep Learning
• Student-Teacher Framework