Linear Reservoir: A Diagonalization-Based Optimization

2026-02-23

AI Summary

This work addresses the computational bottleneck of linear Echo State Networks (Linear ESNs), whose state updates incur O(N²) complexity due to dense matrix multiplication, limiting their applicability in large-scale or real-time settings. To overcome this, the authors reformulate reservoir dynamics in the eigenbasis of the recurrent weight matrix, reducing state propagation to element-wise operations and thereby eliminating costly matrix multiplications. Building on this insight, they propose three diagonalization-based optimization strategies: Eigenvalue Weight Tuning (EWT) to preserve original dynamics, End-to-End Training (EET) of readout weights, and Direct Parameter Generation (DPG) for synthesizing spectral parameters. This paradigm shift enables direct design in the eigenvalue domain, achieving O(N) per-step computational complexity while maintaining competitive prediction accuracy, thus offering an efficient and viable alternative to conventional Linear ESNs.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Evolutionary Computation
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

We introduce a diagonalization-based optimization for Linear Echo State Networks (ESNs) that reduces the per-step computational complexity of reservoir state updates from O(N^2) to O(N). By reformulating reservoir dynamics in the eigenbasis of the recurrent matrix, the recurrent update becomes a set of independent element-wise operations, eliminating the matrix multiplication. We further propose three methods to use our optimization depending on the situation: (i) Eigenbasis Weight Transformation (EWT), which preserves the dynamics of standard and trained Linear ESNs, (ii) End-to-End Eigenbasis Training (EET), which directly optimizes readout weights in the transformed space and (iii) Direct Parameter Generation (DPG), that bypasses matrix diagonalization by directly sampling eigenvalues and eigenvectors, achieving comparable performance than standard Linear ESNs. Across all experiments, both our methods preserve predictive accuracy while offering significant computational speedups, making them a replacement of standard Linear ESNs computations and training, and suggesting a shift of paradigm in linear ESN towards the direct selection of eigenvalues.

Problem

• Linear Echo State Networks
• computational complexity
• reservoir computing
• state updates
• efficiency

Innovation

• Linear Echo State Network
• Diagonalization
• Eigenbasis Optimization
• Computational Complexity Reduction
• Reservoir Computing