Hierarchical Graph Topic Modeling with Topic Tree-based Transformer

2025-02-17

AI Summary

This paper addresses the challenge of jointly modeling topic hierarchies and graph-level structure in hierarchical graph-structured text—where a central document connects to multiple layers of associated documents. We propose the first unified Transformer framework operating in hyperbolic space. Methodologically, we integrate topic tree embeddings with a Hyperbolic Dual Recursive Neural Network (HDRNN), enabling simultaneous encoding of ancestral/sibling tree structures and inter-document graph topology within each Transformer layer, while incorporating dual hierarchical priors (topic and graph). Our key contributions are: (1) joint modeling—within hyperbolic space—of fine-grained intra-document topic hierarchies and long-range inter-document graph dependencies; and (2) a hierarchy-aware self-attention mechanism. Experiments demonstrate significant improvements over baselines on both supervised and unsupervised tasks, accurately capturing topic distributions and cross-layer graph structural relationships.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Topic discovery and tracking

Abstract

Textual documents are commonly connected in a hierarchical graph structure where a central document links to others with an exponentially growing connectivity. Though Hyperbolic Graph Neural Networks (HGNNs) excel at capturing such graph hierarchy, they cannot model the rich textual semantics within documents. Moreover, text contents in documents usually discuss topics of different specificity. Hierarchical Topic Models (HTMs) discover such latent topic hierarchy within text corpora. However, most of them focus on the textual content within documents, and ignore the graph adjacency across interlinked documents. We thus propose a Hierarchical Graph Topic Modeling Transformer to integrate both topic hierarchy within documents and graph hierarchy across documents into a unified Transformer. Specifically, to incorporate topic hierarchy within documents, we design a topic tree and infer a hierarchical tree embedding for hierarchical topic modeling. To preserve both topic and graph hierarchies, we design our model in hyperbolic space and propose Hyperbolic Doubly Recurrent Neural Network, which models ancestral and fraternal tree structure. Both hierarchies are inserted into each Transformer layer to learn unified representations. Both supervised and unsupervised experiments verify the effectiveness of our model.

Problem

• Model hierarchical topic structures in documents
• Integrate graph hierarchy across documents
• Unify topic and graph hierarchies in Transformer

Innovation

• Hierarchical Graph Topic Modeling
• Hyperbolic Doubly Recurrent Neural Network
• Topic Tree-based Transformer