A universal linearized subspace refinement framework for neural networks

2026-01-20

AI Summary

This work addresses the persistent accuracy bottleneck in gradient-trained neural networks, which often fail to achieve their theoretical expressivity due to numerical ill-conditioning induced by the loss function rather than non-convexity. To overcome this limitation, the authors propose the Linearized Subspace Refinement (LSR) framework, which constructs a Jacobian-induced linear residual model around a fixed trained network and solves a subspace-constrained least-squares problem to obtain a high-accuracy linear predictor—without altering the network architecture or training procedure. LSR reveals, for the first time, that numerical ill-conditioning—not optimization landscape non-convexity—is the primary source of the accuracy barrier, offering a universal, one-shot, architecture-agnostic post-processing strategy. Empirical results across function approximation, operator learning, and physics-informed fine-tuning tasks demonstrate up to an order-of-magnitude error reduction and significantly accelerated convergence.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Neural networks are predominantly trained using gradient-based methods, yet in many applications their final predictions remain far from the accuracy attainable within the model's expressive capacity. We introduce Linearized Subspace Refinement (LSR), a general and architecture-agnostic framework that exploits the Jacobian-induced linear residual model at a fixed trained network state. By solving a reduced direct least-squares problem within this subspace, LSR computes a subspace-optimal solution of the linearized residual model, yielding a refined linear predictor with substantially improved accuracy over standard gradient-trained solutions, without modifying network architectures, loss formulations, or training procedures. Across supervised function approximation, data-driven operator learning, and physics-informed operator fine-tuning, we show that gradient-based training often fails to access this attainable accuracy, even when local linearization yields a convex problem. This observation indicates that loss-induced numerical ill-conditioning, rather than nonconvexity or model expressivity, can constitute a dominant practical bottleneck. In contrast, one-shot LSR systematically exposes accuracy levels not fully exploited by gradient-based training, frequently achieving order-of-magnitude error reductions. For operator-constrained problems with composite loss structures, we further introduce Iterative LSR, which alternates one-shot LSR with supervised nonlinear alignment, transforming ill-conditioned residual minimization into numerically benign fitting steps and yielding accelerated convergence and improved accuracy. By bridging nonlinear neural representations with reduced-order linear solvers at fixed linearization points, LSR provides a numerically grounded and broadly applicable refinement framework for supervised learning, operator learning, and scientific computing.

Problem

• neural networks
• gradient-based training
• numerical ill-conditioning
• accuracy gap
• expressive capacity

Innovation

• Linearized Subspace Refinement
• Jacobian-induced linearization
• numerical ill-conditioning
• operator learning
• least-squares refinement