Tensor-to-Tensor Models with Fast Iterated Sum Features

2025-06-06

AI Summary

To address the urgent need for sub-quadratic computational complexity in high-dimensional tensor data processing (e.g., images), this paper introduces FIS—the first linear-complexity, plug-and-play tensor-to-tensor layer. Methodologically, we pioneer the extension of multi-parameter iteration and signature theory to higher-order tensors, integrating Corner Trees for geometric structure modeling and tensor decomposition for efficient approximation—yielding a differentiable, geometry-aware layer with state-space generalization and seamless compatibility with architectures like ResNet. Contributions include: (1) the first sub-quadratic tensor layer, breaking classical tensor operation complexity bottlenecks; (2) achieving ResNet-level accuracy on image classification with significantly fewer parameters (only +0.1% error); and (3) attaining 97.3% AUROC on MVTec AD texture anomaly detection. The implementation is fully open-sourced.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Computer Vision: Other Foundations of Computer Vision
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Data in the form of images or higher-order tensors is ubiquitous in modern deep learning applications. Owing to their inherent high dimensionality, the need for subquadratic layers processing such data is even more pressing than for sequence data. We propose a novel tensor-to-tensor layer with linear cost in the input size, utilizing the mathematical gadget of ``corner trees'' from the field of permutation counting. In particular, for order-two tensors, we provide an image-to-image layer that can be plugged into image processing pipelines. On the one hand, our method can be seen as a higher-order generalization of state-space models. On the other hand, it is based on a multiparameter generalization of the signature of iterated integrals (or sums). The proposed tensor-to-tensor concept is used to build a neural network layer called the Fast Iterated Sums (FIS) layer which integrates seamlessly with other layer types. We demonstrate the usability of the FIS layer with both classification and anomaly detection tasks. By replacing some layers of a smaller ResNet architecture with FIS, a similar accuracy (with a difference of only 0.1%) was achieved in comparison to a larger ResNet while reducing the number of trainable parameters and multi-add operations. The FIS layer was also used to build an anomaly detection model that achieved an average AUROC of 97.3% on the texture images of the popular MVTec AD dataset. The processing and modelling codes are publicly available at https://github.com/diehlj/fast-iterated-sums.

Problem

• Develops subquadratic tensor-to-tensor layers for high-dimensional data
• Generalizes state-space models for higher-order tensor processing
• Enables efficient image classification and anomaly detection tasks

Innovation

• Linear-cost tensor-to-tensor layer using corner trees
• Higher-order generalization of state-space models
• Multiparameter generalization of iterated sums signature