Algebraformer: A Neural Approach to Linear Systems

2025-11-18

AI Summary

To address the limited stability and generalizability of existing solvers for ill-conditioned linear systems—stemming from reliance on handcrafted preconditioners, parameter tuning, and domain-specific knowledge—this paper proposes Algebraformer, an end-to-end neural solver based on the Transformer architecture. Its core innovation is a lightweight matrix encoding scheme that directly models algebraic relationships between the coefficient matrix and right-hand-side vector in *O*(*n*²) memory complexity, eliminating conventional preprocessing and manual intervention. Evaluated on scientific computing tasks—including spectral method interpolation and Newton-type acceleration—Algebraformer achieves accuracy comparable to classical numerical methods while significantly reducing inference latency. Extensive experiments demonstrate both the effectiveness and scalability of this generic neural architecture for solving ill-conditioned systems, establishing a data-driven paradigm for scientific computing.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Cognitive Modeling & Cognitive Systems: (Computational) Cognitive Architectures
• Multiagent Systems: Distributed Problem Solving

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Recent work in deep learning has opened new possibilities for solving classical algorithmic tasks using end-to-end learned models. In this work, we investigate the fundamental task of solving linear systems, particularly those that are ill-conditioned. Existing numerical methods for ill-conditioned systems often require careful parameter tuning, preconditioning, or domain-specific expertise to ensure accuracy and stability. In this work, we propose Algebraformer, a Transformer-based architecture that learns to solve linear systems end-to-end, even in the presence of severe ill-conditioning. Our model leverages a novel encoding scheme that enables efficient representation of matrix and vector inputs, with a memory complexity of $O(n^2)$, supporting scalable inference. We demonstrate its effectiveness on application-driven linear problems, including interpolation tasks from spectral methods for boundary value problems and acceleration of the Newton method. Algebraformer achieves competitive accuracy with significantly lower computational overhead at test time, demonstrating that general-purpose neural architectures can effectively reduce complexity in traditional scientific computing pipelines.

Problem

• Solving ill-conditioned linear systems with neural networks
• Reducing computational overhead in scientific computing tasks
• Eliminating parameter tuning for linear system solvers

Innovation

• Transformer-based architecture for solving linear systems
• Novel encoding scheme with O(n^2) memory complexity
• Competitive accuracy with lower computational overhead