Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theory

2024-05-26

arXiv.org

AI Summary

This work identifies fundamental computational limitations of Structured State Space Models (SSMs) and Transformers in complex reasoning and function composition tasks. Method: Leveraging formal language theory, circuit complexity analysis, and constructive proofs, the study rigorously characterizes the expressive power and scalability constraints of these architectures. Contribution/Results: It establishes—for the first time—that single-layer SSMs cannot efficiently implement long-range function composition; their finite-precision languages recognize only regular languages. Moreover, Chain-of-Thought prompting fails to overcome the intrinsic exponential blowup in required reasoning steps as problem size grows. Empirical evaluation across multi-digit multiplication, dynamic programming, and the Einstein puzzle reveals severe performance degradation in both models, with strong reliance on spurious shortcuts and error accumulation. The work formally delineates the computational ceiling of current dominant architectures, providing critical theoretical foundations and benchmarking criteria for advancing robust multi-step reasoning and general artificial intelligence.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Despite their successes, deep learning models struggle with tasks requiring complex reasoning and function composition. We present a theoretical and empirical investigation into the limitations of Structured State Space Models (SSMs) and Transformers in such tasks. We prove that one-layer SSMs cannot efficiently perform function composition over large domains without impractically large state sizes, and even with Chain-of-Thought prompting, they require a number of steps that scale unfavorably with the complexity of the function composition. Also, the language of a finite-precision SSM is within the class of regular languages. Our experiments corroborate these theoretical findings. Evaluating models on tasks including various function composition settings, multi-digit multiplication, dynamic programming, and Einstein's puzzle, we find significant performance degradation even with advanced prompting techniques. Models often resort to shortcuts, leading to compounding errors. These findings highlight fundamental barriers within current deep learning architectures rooted in their computational capacities. We underscore the need for innovative solutions to transcend these constraints and achieve reliable multi-step reasoning and compositional task-solving, which is critical for advancing toward general artificial intelligence.

Problem

• Deep learning models struggle with complex reasoning tasks.
• SSMs and Transformers face limitations in function composition.
• Current architectures need innovation for reliable multi-step reasoning.

Innovation

• Analyzed SSMs and Transformers limitations theoretically
• Empirically tested models on complex reasoning tasks
• Highlighted need for new deep learning solutions