Neural networks leverage nominally quantum and post-quantum representations

2025-07-10

AI Summary

This study investigates how deep neural networks (Transformers/RNNs) spontaneously acquire and represent low-dimensional “quantum” and “post-quantum” structures—reflecting the underlying data-generating process—during standard next-token prediction pretraining. Leveraging geometric analysis of neural activation spaces, we demonstrate that these networks implicitly perform Bayesian iterative updates over latent world-model states: activation patterns induced by historical inputs encode probability densities over future token distributions. Crucially, we provide the first empirical evidence that pretrained networks capture quantum-level representational structures inaccessible to classical circuits, with remarkable architectural consistency across Transformers and RNNs. Our results indicate that pretraining inherently induces efficient, robust world models that transcend classical computational limits. This work establishes a novel paradigm for understanding the inductive biases and reasoning mechanisms of deep learning. (149 words)

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: (Large) Language Models
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

We show that deep neural networks, including transformers and RNNs, pretrained as usual on next-token prediction, intrinsically discover and represent beliefs over 'quantum' and 'post-quantum' low-dimensional generative models of their training data -- as if performing iterative Bayesian updates over the latent state of this world model during inference as they observe more context. Notably, neural nets easily find these representation whereas there is no finite classical circuit that would do the job. The corresponding geometric relationships among neural activations induced by different input sequences are found to be largely independent of neural-network architecture. Each point in this geometry corresponds to a history-induced probability density over all possible futures, and the relative displacement of these points reflects the difference in mechanism and magnitude for how these distinct pasts affect the future.

Problem

• Neural networks discover quantum-like generative models
• No classical circuit achieves these representations effectively
• Geometric relationships in activations are architecture-independent

Innovation

• Neural networks discover quantum-like representations
• Transformers and RNNs perform Bayesian updates
• Geometric relationships independent of architecture