Revisiting Random Walks for Learning on Graphs

2024-07-01

arXiv.org

AI Summary

This work addresses the lack of rigorous graph isomorphism invariance and probabilistic universal approximation capability in existing graph neural networks (GNNs). We propose Random Walk Neural Networks (RWNNs), which treat anonymized random walk sequences on graphs as textual inputs to language models (e.g., Transformers) for vertex- and graph-level prediction. Theoretically, we prove that any anonymized random walk trajectory satisfies isomorphism invariance in a probabilistic sense; further, we show RWNNs inherently mitigate oversmoothing and reinterpret overcompression as probabilistic undercoverage. Our method integrates Markov chain analysis, graph anonymization encoding, and contrastive message-passing modeling. Experiments demonstrate that RWNNs surpass the 3-WL test on challenging regular graph discrimination tasks and achieve state-of-the-art performance on transductive node classification in the arXiv citation network. Both theoretical analysis and empirical evaluation consistently validate RWNNs’ isomorphism invariance, expressive power, and generalization superiority.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Reasoning under Uncertainty: Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Computational social science
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

We revisit a simple model class for machine learning on graphs, where a random walk on a graph produces a machine-readable record, and this record is processed by a deep neural network to directly make vertex-level or graph-level predictions. We call these stochastic machines random walk neural networks (RWNNs), and through principled analysis, show that we can design them to be isomorphism invariant while capable of universal approximation of graph functions in probability. A useful finding is that almost any kind of record of random walks guarantees probabilistic invariance as long as the vertices are anonymized. This enables us, for example, to record random walks in plain text and adopt a language model to read these text records to solve graph tasks. We further establish a parallelism to message passing neural networks using tools from Markov chain theory, and show that over-smoothing in message passing is alleviated by construction in RWNNs, while over-squashing manifests as probabilistic under-reaching. We empirically demonstrate RWNNs on a range of problems, verifying our theoretical analysis and demonstrating the use of language models for separating strongly regular graphs where 3-WL test fails, and transductive classification on arXiv citation network. Code is available at https://github.com/jw9730/random-walk.

Problem

• Develops random walk neural networks for graph learning tasks.
• Ensures isomorphism invariance and universal graph function approximation.
• Addresses over-smoothing and over-squashing in graph neural networks.

Innovation

• Random walk neural networks for graph learning
• Language models process anonymized random walk records
• Alleviates over-smoothing in message passing neural networks