Approximation and learning with compositional tensor trains

2025-12-19

AI Summary

This paper addresses the longstanding trade-off between expressive power and computational efficiency in multivariate function approximation. We propose Compositional Tensor Trains (CTTs), a novel framework that embeds tensor trains within a compositional architecture, enabling unified modeling of sparse polynomials, constant-width deep neural networks (DNNs), and general tensor networks via hierarchical composition of low-rank functions. To optimize CTTs, we introduce a dual-path algorithm integrating natural gradient descent with optimal control principles, coupled with low-rank Gram matrix estimation and tensor random sketching to enable intra-layer controllable compression. Theoretically grounded and empirically validated, CTTs achieve accuracy on par with deep neural networks on regression tasks while offering strong interpretability, linear parameter complexity O(n), and efficient training. Our approach establishes a new paradigm for high-dimensional function learning—balancing expressivity, scalability, and analytical tractability.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We introduce compositional tensor trains (CTTs) for the approximation of multivariate functions, a class of models obtained by composing low-rank functions in the tensor-train format. This format can encode standard approximation tools, such as (sparse) polynomials, deep neural networks (DNNs) with fixed width, or tensor networks with arbitrary permutation of the inputs, or more general affine coordinate transformations, with similar complexities. This format can be viewed as a DNN with width exponential in the input dimension and structured weights matrices. Compared to DNNs, this format enables controlled compression at the layer level using efficient tensor algebra. On the optimization side, we derive a layerwise algorithm inspired by natural gradient descent, allowing to exploit efficient low-rank tensor algebra. This relies on low-rank estimations of Gram matrices, and tensor structured random sketching. Viewing the format as a discrete dynamical system, we also derive an optimization algorithm inspired by numerical methods in optimal control. Numerical experiments on regression tasks demonstrate the expressivity of the new format and the relevance of the proposed optimization algorithms. Overall, CTTs combine the expressivity of compositional models with the algorithmic efficiency of tensor algebra, offering a scalable alternative to standard deep neural networks.

Problem

• Introduces compositional tensor trains for approximating multivariate functions
• Enables controlled compression at layer level using efficient tensor algebra
• Offers a scalable alternative to standard deep neural networks

Innovation

• Compositional tensor trains approximate multivariate functions
• Layerwise algorithm uses natural gradient descent inspiration
• Combines compositional expressivity with tensor algebra efficiency