How AI settled the complexity of the oldest SGD algorithm

2026-06-28

AI Summary

This study resolves the long-standing open problem of the worst-case time complexity of the Kaczmarz algorithm—the earliest known stochastic gradient descent method—since its introduction in 1937. For the first time, we integrate the reasoning capabilities of large language models with rigorous tools from theoretical computer science to model and formally verify the complexity of this classical optimization algorithm. Our work establishes a tight upper bound on the worst-case complexity of the Kaczmarz algorithm and pioneers a novel paradigm that leverages modern artificial intelligence to collaboratively derive theoretical properties of classical numerical methods. This breakthrough provides fundamental insights into the theoretical underpinnings of stochastic gradient descent–type algorithms.

Abstract

In 1937, Stefan Kaczmarz proposed a simple algorithm for solving systems of linear equations. This algorithm turned out to be the earliest known example of stochastic gradient descent, a ubiquitous computing paradigm that drives the training of modern AI models such as ChatGPT and Gemini. Now, those AI models have joined forces to discover the worst-case complexity of the Kaczmarz algorithm. This paper tells the story of how it happened.

Problem

• Kaczmarz algorithm
• stochastic gradient descent
• worst-case complexity
• linear equations
• AI

Innovation

• Kaczmarz algorithm
• stochastic gradient descent
• worst-case complexity
• AI-assisted discovery
• linear systems