On Powerful Ways to Generate: Autoregression, Diffusion, and Beyond

2025-10-07

AI Summary

Existing generative paradigms—such as autoregressive and masked diffusion models—exhibit limitations in flexibility, editability, and cross-domain adaptability. To address these, this paper proposes an architecture-agnostic, abstract modeling framework for the generation process. Its core innovation is a novel generative mechanism supporting **rewritability** and **variable-length editing**, thereby overcoming constraints inherent in unidirectional prediction or fixed-step sampling. Through formal analysis of computational complexity and learnability, we theoretically establish that this paradigm achieves superior expressive power and faster convergence. Empirical evaluation demonstrates significant improvements in accuracy, controllable editing, and domain generalization across challenging tasks—including program synthesis and scientific reasoning—while offering a unified, scalable foundation for structured generation beyond natural language. (149 words)

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This paper formally studies generation processes, including auto-regressive next-token prediction and masked diffusion, that abstract beyond architectural specifics. At this level of abstraction, we quantify their benefits and limitations through measurable criteria such as computational hardness and learnability. In particular, we demonstrate that allowing generation to proceed beyond autoregression and current masked diffusion, with capabilities to rewrite and length-variable edit, can bring significant theoretical and empirical advantages, with important implications for frontier LLMs that aspire to tackle increasingly hard problems and work universally across domains beyond natural language, such as coding and science.

Problem

• Formally studies generation processes like autoregression and diffusion
• Quantifies computational hardness and learnability of generation methods
• Demonstrates advantages of rewrite-capable generation for LLMs

Innovation

• Autoregressive and diffusion generation processes abstractly studied
• Quantified benefits via computational hardness and learnability criteria
• Extended generation with rewrite and variable-length edit capabilities