Learning-Order Autoregressive Models with Application to Molecular Graph Generation

2025-03-07

AI Summary

Graph-structured data (e.g., molecular graphs) lack a natural autoregressive generation order, hindering sequential generative modeling. Method: This paper proposes a learnable, dynamic ordering strategy that models the generation sequence as a state-dependent probability distribution, jointly optimized with the graph generation process. It innovatively formulates the autoregressive ordering as a trainable policy and establishes an end-to-end optimization framework grounded in the variational lower bound, integrating variational inference, gradient estimation (e.g., REINFORCE or Gumbel-Softmax), and graph neural networks to enable co-learning of ordering and structure. Contribution/Results: The approach achieves state-of-the-art performance on molecular generation benchmarks QM9 and ZINC250k, significantly surpassing prior methods in Fréchet Chemical Distance (FCD). It is the first method to realize fully data-driven, jointly optimized learning of generation order—without relying on handcrafted heuristics or fixed traversal rules.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Autoregressive models (ARMs) have become the workhorse for sequence generation tasks, since many problems can be modeled as next-token prediction. While there appears to be a natural ordering for text (i.e., left-to-right), for many data types, such as graphs, the canonical ordering is less obvious. To address this problem, we introduce a variant of ARM that generates high-dimensional data using a probabilistic ordering that is sequentially inferred from data. This model incorporates a trainable probability distribution, referred to as an emph{order-policy}, that dynamically decides the autoregressive order in a state-dependent manner. To train the model, we introduce a variational lower bound on the exact log-likelihood, which we optimize with stochastic gradient estimation. We demonstrate experimentally that our method can learn meaningful autoregressive orderings in image and graph generation. On the challenging domain of molecular graph generation, we achieve state-of-the-art results on the QM9 and ZINC250k benchmarks, evaluated using the Fr'{e}chet ChemNet Distance (FCD).

Problem

• Autoregressive models lack natural ordering for graph data.
• Introduces a variant of ARM with probabilistic ordering for high-dimensional data.
• Achieves state-of-the-art results in molecular graph generation benchmarks.

Innovation

• Autoregressive models with probabilistic ordering
• Trainable order-policy for dynamic sequence generation
• Variational lower bound optimization for training