🤖 AI Summary
This work addresses the high computational and memory overhead of Vision Transformers in deployment by introducing, for the first time, the tensor cosine product (C-product) into the Vision Transformer architecture. Leveraging the multilinear structure inherent in images and the orthogonality of the cosine transform, the proposed method constructs an efficient attention mechanism and structured feature representation. It achieves competitive accuracy while reducing the number of parameters to 1/C of the original model, where C denotes the number of channels, thereby substantially lowering model complexity. By integrating multilinear algebra with orthogonal transforms, this approach makes notable contributions both theoretically and in terms of practical deployment efficiency.
📝 Abstract
Vision Transformers have achieved state-of-the-art performance in a wide range
of computer vision tasks, but their practical deployment is limited by high
computational and memory costs. In this paper, we introduce a novel tensor-based
framework for Vision Transformers built upon the Tensor Cosine Product
(Cproduct). By exploiting multilinear structures inherent in image data and the
orthogonality of cosine transforms, the proposed approach enables efficient
attention mechanisms and structured feature representations. We develop the
theoretical foundations of the tensor cosine product, analyze its algebraic
properties, and integrate it into a new Cproduct-based Vision Transformer
architecture (TCP-ViT). Numerical experiments on standard classification and
segmentation benchmarks demonstrate that the proposed method achieves a uniform
1/C parameter reduction (where C is the number of channels) while
maintaining competitive accuracy.