Position: We Need An Algorithmic Understanding of Generative AI

2025-07-10

AI Summary

There is a lack of theoretical and empirical understanding of the actual algorithms learned and executed by large language models (LLMs), with existing research predominantly focusing on scaling laws while neglecting internal algorithmic mechanisms. Method: This project introduces AlgEval—a novel framework for algorithm-level analysis of generative AI—employing a dual-path approach: hypothesis-driven modeling coupled with circuit-level verification, integrated with fine-grained analysis of attention patterns and hidden-state dynamics. Contribution/Results: AlgEval systematically identifies emergent algorithmic primitives—including search, recursion, and their compositional logic—during model inference. By advancing interpretability from phenomenological description to mechanistic explanation, it provides empirically grounded foundations and methodological tools for efficient training, performance optimization, and multi-agent system design.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Algorithm Configuration

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

What algorithms do LLMs actually learn and use to solve problems? Studies addressing this question are sparse, as research priorities are focused on improving performance through scale, leaving a theoretical and empirical gap in understanding emergent algorithms. This position paper proposes AlgEval: a framework for systematic research into the algorithms that LLMs learn and use. AlgEval aims to uncover algorithmic primitives, reflected in latent representations, attention, and inference-time compute, and their algorithmic composition to solve task-specific problems. We highlight potential methodological paths and a case study toward this goal, focusing on emergent search algorithms. Our case study illustrates both the formation of top-down hypotheses about candidate algorithms, and bottom-up tests of these hypotheses via circuit-level analysis of attention patterns and hidden states. The rigorous, systematic evaluation of how LLMs actually solve tasks provides an alternative to resource-intensive scaling, reorienting the field toward a principled understanding of underlying computations. Such algorithmic explanations offer a pathway to human-understandable interpretability, enabling comprehension of the model's internal reasoning performance measures. This can in turn lead to more sample-efficient methods for training and improving performance, as well as novel architectures for end-to-end and multi-agent systems.

Problem

• Understanding algorithms learned by LLMs for problem-solving
• Developing AlgEval to study latent representations and attention
• Enabling human-interpretable explanations of model reasoning

Innovation

• Proposes AlgEval framework for LLM algorithms
• Analyzes latent representations and attention patterns
• Combines top-down hypotheses with bottom-up tests