Domain-Agnostic Neural Topic Modeling with Contextual Token-Level Semantic Graph Representation

2026-08-17

AI Summary

This study addresses the degradation of topic interpretability and geometric distortion in embedding spaces when applying pretrained language models to specialized corpora. We propose DARTopic, a novel framework that constructs token-level semantic graphs and jointly trains graph neural networks with topic inference to restructure the embedding space without fine-tuning the encoder, thereby overcoming the capacity bottleneck of frozen encoders. Experiments across general, biomedical, and legal benchmarks demonstrate that DARTopic outperforms strong baselines, significantly enhancing cross-domain topic coherence and clustering performance. The framework exhibits both robustness and efficiency, achieving high-quality, domain-agnostic topic modeling while preserving the semantic integrity of pretrained representations without additional parameter updates.

Abstract

Recent advances in neural topic models with pre-trained language models (PLMs) have achieved strong performance by leveraging general-domain pre-training, yet their topic interpretability often degrades on specialized corpora. This limitation primarily stems from the geometry of the embedding space, where domain-specific terms unseen during pre-training collapse into an indistinguishable region, and neither domain-specific re-training, word-level graph enrichment, nor parameter-efficient fine-tuning can restructure this space without inheriting the capacity ceiling of the underlying encoder. Our key insight is that a learnable graph layer operating on token-level PLM embeddings can acquire corpus-specific semantic structure that the frozen encoder lacks, because token-level graphs preserve document-local context that word-level representations discard and joint optimization with the topic objective reshapes embedding geometry directly from target-domain evidence. We instantiate this insight as DARTopic, a domain-agnostic framework that constructs token-level semantic graphs from frozen PLM embeddings and jointly trains a GNN encoder with topic inference. Across three benchmarks spanning general, biomedical, and legal domains, DARTopic consistently outperforms strong baselines in topic coherence and document clus- tering without any encoder fine-tuning, while demonstrating robustness to PLM choice and favorable runtime efficiency over fine-tuning based alternatives.

Problem

• Neural Topic Modeling
• Topic Interpretability
• Domain-Specific Corpora
• Embedding Space Geometry
• Pre-trained Language Models

Innovation

• Token-Level Semantic Graph
• Domain-Agnostic Neural Topic Modeling
• Frozen PLM Embeddings
• Joint Optimization
• Embedding Geometry Reshaping