Hybrid Least Squares/Gradient Descent Methods for MIONets

2026-07-07

AI Summary

This work addresses the inefficiency in training MIONets, which arises from parameter coupling between multiple branches and the trunk network. The authors propose a hybrid least-squares/gradient descent (LSGD) method, extending it for the first time to MIONets by treating the network as a multilinear function of the parameters in the final layers of each branch. Branch parameters are efficiently updated via alternating least squares, while the remaining parameters are optimized using gradient descent. The approach leverages Kronecker products, Khatri-Rao products, and tensor permutation matrices to decompose large-scale linear systems, enabling efficient handling of general L² losses with regularization terms and linear operators. Experiments demonstrate that the method significantly improves training efficiency, accelerates convergence without compromising accuracy, and remains compatible with diverse loss structures.

Abstract

In this paper, we propose an efficient hybrid least squares/gradient descent (LSGD) method for MIONets to accelerate training. This method generalizes the LSGD method for DeepONets. Since MIONet is the sum of the entrywise product of multiple branch networks and a trunk network, it can be viewed as a multilinear function with respect to the last layer parameters of each branch network. These sets of parameters can be optimized using the alternating least squares method, where we solve the LS system for a single branch network in turn. To handle the large-sized system matrix, we introduce Kronecker and Khatri-Rao products and tensor permutation matrices to factor the large matrix into small ones. Our method is compatible with a general type of $L^2$ loss with regularization terms for the last layer parameters of each branch, where linear operators can be applied to the MIONet output in each loss term.

Problem

• MIONets
• training acceleration
• least squares
• gradient descent
• alternating optimization

Innovation

• MIONets
• hybrid least squares/gradient descent
• alternating least squares
• Kronecker product
• multilinear optimization