Flatten Graphs as Sequences: Transformers are Scalable Graph Generators

2025-02-04

AI Summary

To address the structural complexity and limited scalability in large-scale attributed graph generation, this paper proposes AutoGraph: a novel framework featuring the first reversible, edge-centric graph flattening mechanism that losslessly maps graphs to sequences. It employs a decoder-only Transformer for autoregressive graph generation without requiring node features. The method enables substructure-conditional generation without fine-tuning, establishing a new paradigm for graph foundation models. Evaluated on synthetic and molecular graph benchmarks, AutoGraph achieves state-of-the-art performance—accelerating generation by 100× and training by 3× compared to prior methods—while supporting cross-domain transfer and controllable substructure generation.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Natural Language Processing: Generation
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We introduce AutoGraph, a novel autoregressive framework for generating large attributed graphs using decoder-only transformers. At the core of our approach is a reversible"flattening"process that transforms graphs into random sequences. By sampling and learning from these sequences, AutoGraph enables transformers to model and generate complex graph structures in a manner akin to natural language. In contrast to diffusion models that rely on computationally intensive node features, our approach operates exclusively on these sequences. The sampling complexity and sequence length scale linearly with the number of edges, making AutoGraph highly scalable for generating large sparse graphs. Empirically, AutoGraph achieves state-of-the-art performance across diverse synthetic and molecular graph generation benchmarks, while delivering a 100-fold generation and a 3-fold training speedup compared to leading diffusion models. Additionally, it demonstrates promising transfer capabilities and supports substructure-conditioned generation without additional fine-tuning. By extending language modeling techniques to graph generation, this work paves the way for developing graph foundation models.

Problem

• Autoregressive framework for graph generation
• Transformers model complex graph structures
• Scalable generation of large sparse graphs

Innovation

• Autoregressive framework for graph generation
• Transformers model complex graph structures
• Linear scalability with edge number