OccamVTS: Distilling Vision Models to 1% Parameters for Time Series Forecasting

2025-08-03

AI Summary

Large vision models exhibit parameter redundancy and introduce noisy high-level semantic features when adapted to time-series forecasting. Method: This paper proposes OccamVTS, a knowledge distillation framework leveraging a pre-trained vision model as a “privileged teacher.” Through cross-modal analysis, it reveals that time-series forecasting relies predominantly on low-level texture features; accordingly, OccamVTS introduces pyramid-style feature alignment and correlation-aware distillation to selectively transfer only the most discriminative 1% of parameters to a lightweight student network. Contribution/Results: Counterintuitively, aggressive parameter pruning (99% reduction) improves accuracy and significantly enhances generalization in few-shot and zero-shot settings. On multiple benchmarks, OccamVTS achieves state-of-the-art performance using only 1% of the parameters—delivering superior prediction accuracy, minimal computational overhead, and strong robustness.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Computer Vision: Diffusion Models for Vision
• Data Mining & Knowledge Management: Data Compression

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Time series forecasting is fundamental to diverse applications, with recent approaches leverage large vision models (LVMs) to capture temporal patterns through visual representations. We reveal that while vision models enhance forecasting performance, 99% of their parameters are unnecessary for time series tasks. Through cross-modal analysis, we find that time series align with low-level textural features but not high-level semantics, which can impair forecasting accuracy. We propose OccamVTS, a knowledge distillation framework that extracts only the essential 1% of predictive information from LVMs into lightweight networks. Using pre-trained LVMs as privileged teachers, OccamVTS employs pyramid-style feature alignment combined with correlation and feature distillation to transfer beneficial patterns while filtering out semantic noise. Counterintuitively, this aggressive parameter reduction improves accuracy by eliminating overfitting to irrelevant visual features while preserving essential temporal patterns. Extensive experiments across multiple benchmark datasets demonstrate that OccamVTS consistently achieves state-of-the-art performance with only 1% of the original parameters, particularly excelling in few-shot and zero-shot scenarios.

Problem

• Reducing vision model parameters for time series forecasting
• Filtering irrelevant visual features to improve accuracy
• Achieving high performance with minimal model parameters

Innovation

• Distills vision models to 1% parameters
• Uses pyramid-style feature alignment
• Filters semantic noise via correlation distillation