S+t-SNE - Bringing dimensionality reduction to data streams

2024-03-26

International Symposium on Intelligent Data Analysis

AI Summary

To address the challenge of real-time dimensionality reduction and visualization for dynamic data streams, this paper proposes the first strictly single-pass streaming t-SNE algorithm. Methodologically, it introduces a sliding-window optimization framework coupled with a similarity reweighting mechanism, integrating incremental gradient updates, dynamic neighborhood maintenance, adaptive learning rates, and online reconstruction of sparse similarity matrices—balancing local structure preservation and temporal efficiency. The core contribution is the first extension of t-SNE to a rigorously single-pass streaming paradigm, enabling continuous embedding updates and millisecond-level incorporation of new samples. Experiments on diverse real-world data streams demonstrate that the method achieves over 90% of offline t-SNE’s dimensional reduction quality, reduces inference latency by two orders of magnitude, and maintains constant memory footprint—substantially outperforming existing incremental or approximate alternatives.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Data Mining & Knowledge Management: Data Stream Mining
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Problem

• Data Streaming
• Visualization
• Real-time Analysis

Innovation

• S+t-SNE
• Continuous Data Streaming
• Real-time Pattern Discovery