On Dimension-Free Transformer: An Application of STP to AI

2025-04-20

AI Summary

This work addresses the limitation of conventional Transformers—fixed input/output dimensions—that hinders their application to multidimensional signal processing. We propose the first dimension-free Transformer framework, whose core innovation is the introduction of the semi-tensor product (STP) into Transformer modeling. Specifically, we design Projection-Based Tensor Hypervector Transformation (PBTH), a projection-driven linear hypervector transformation mechanism that seamlessly replaces all linear modules in standard Transformers (e.g., attention projections and feed-forward networks). PBTH eliminates reliance on fixed embedding dimensions, enabling arbitrary-dimensional inputs and outputs while preserving balanced information representation across dimensions. Extensive experiments demonstrate that PBTH significantly improves modeling efficiency and generalization performance on multidimensional signal tasks. Our approach establishes a new paradigm for extending Transformer architectures to high-dimensional and heterogeneous signal processing.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Data transparency and provenance

Abstract

The matrix expressions for every parts of a transformer are firstly described. Based on semi-tensor product (STP) of matrices the hypervectors are reconsidered and the linear transformation over hypervectors is constructed by using projection. Its properties and calculating formulas are obtained. Using projection-based transformation of hypervector (PBTH), the framework of dimension-free transformer (DFT) is proposed by verifying each linear transformation in a transformer and replacing it by a proper PBTH, which allows the inputs and outputs being of arbitrary dimensions. Using balanced information about all entries, DFT must be more efficient in dealing with signals.

Problem

• Develops matrix expressions for transformer components
• Proposes dimension-free transformer using STP
• Enables arbitrary input-output dimensions efficiently

Innovation

• Matrix expressions for transformer parts
• Semi-tensor product linear transformation
• Dimension-free framework via PBTH