Accelerated Learning of High Dimensional Functions with a Tensor-Featured Training Network

2026-08-10

AI Summary

This work addresses the challenges of slow optimization and excessive memory consumption in deep neural networks (DNNs) for high-dimensional function learning by proposing an efficient framework that integrates contextual features with tensor network decomposition. The approach introduces rapidly computable rank-1 features alongside expressive tensor-network-based features at the first layer of the DNN. By alternately optimizing network parameters and the feature layer—augmented with discretization, feature distillation, and randomized tensor decomposition strategies—the model representation is substantially compressed. Experimental results demonstrate that the proposed method reduces memory requirements by at least eight orders of magnitude, enabling efficient training of functions ranging from 5 to 40 dimensions while maintaining strong computational efficiency and representational capacity.

Abstract

In this work we present a method to accelerate the optimization of learning high dimensional functions using deep neural network (DNN). This optimization procedure introduces contextual features into the first layer of a DNN. The parameters of DNN are optimized via standard gradient descent while keeping the input-feature basis fixed. After optimization of the DNN parameters, the feature layer is provided a chance to update and change before DNN optimization resumes. The feature layer has two types of functions: those that can be evaluated quickly in a matrix-free way on the domain (i.e. rank-1 features) and more complex features that must first be decomposed using tensor network (TN) decomposition strategies (tensor features). In particular, we study the effect of adding features which distill pretrained DNN into TNs using a discretize and decompose strategy. To efficiently decompose high-dimensional functions constructed from discretized DNN, we leverage a randomized tensor decomposition strategy. Using randomization, we are able to reduce the storage cost of decomposing high dimensional functions by at least 8 orders of magnitude. Using this approach, we are able to efficiently train models between 5 and 40 dimensions.

Problem

• high dimensional functions
• accelerated learning
• tensor decomposition
• deep neural network
• optimization

Innovation

• tensor network
• randomized tensor decomposition
• high-dimensional function approximation
• feature layer optimization
• discretize-and-decompose