NAG: A Unified Native Architecture for Encoder-free Text-Graph Modeling in Language Models

2026-01-30

AI Summary

This work addresses the limitation of existing approaches that rely on external graph neural networks to incorporate graph structure into language models, often resulting in a disconnection between semantic and topological information and requiring complex alignment mechanisms. To overcome this, we propose the NAG framework, which for the first time internalizes graph structure modeling directly within the language model itself. By reformulating the self-attention mechanism and recalibrating positional encoding, NAG jointly processes textual semantics and graph topology without any external encoder. We present two efficient instantiations—NAG-Zero and NAG-LoRA—and demonstrate their native graph understanding capabilities across diverse graph-related tasks. The resulting models are not only more compact but also achieve superior performance, effectively establishing intrinsic consistency between semantic content and structural information.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Prevailing methods for integrating graphs into Language Models (LMs) typically rely on a segregated architecture: external Graph Neural Networks (GNNs) encode structural topology, while LMs process textual semantics. We argue this approach is suboptimal for text-graphs: it creates a conceptually disjointed interaction paradigm. By segregating structural encoding from semantic processing, these systems must perform a complex implicit alignment between abstract graph tokens and concrete textual elements. Challenging the necessity of external encoders, we propose NAG (Native Architecture for Graphs), a unified framework that internalizes graph processing within the LM's native manifold. Instead of bridging disparate embedding spaces, NAG repurposes the self-attention mechanism to enforce topological dependencies and recalibrates positional IDs to ensure structural equivalence. This allows the model to harness its intrinsic linguistic capability to simultaneously comprehend node and edge content alongside structural topology. We introduce two efficient implementations: NAG-Zero for absolute preservation of the base model's linguistic capabilities, and NAG-LoRA for enhanced structural adaptation. Experiments across diverse graph tasks validate that NAG achieves robust graph comprehension without the overhead of external encoders, offering a simpler, more coherent paradigm for text-graph modeling.

Problem

• text-graph modeling
• language models
• graph neural networks
• structural alignment
• encoder-free

Innovation

• native graph modeling
• encoder-free architecture
• self-attention adaptation
• structural equivalence
• language model integration