Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scaling

2026-04-23

AI Summary

Large token embedding tables suffer from low parameter efficiency, high memory overhead, under-trained long-tail tokens, and embedding redundancy. This work proposes the X-GRAM framework, which introduces a frequency-aware dynamic token injection mechanism, compresses long-tail tokens via hybrid hashing and alias aliasing, and extracts local n-gram features using normalized SwiGLU ShortConv. These features are fused into both the attention value stream and residual path through depth-aware gating, aligning static memory with dynamic context. X-GRAM pioneers a memory-centric expansion axis that decouples model capacity from computational cost. Evaluated on 0.73B and 1.15B models, X-GRAM achieves an average accuracy gain of 4.4 points over the original backbone and outperforms strong retrieval baselines by 3.2 points, maintaining significant superiority even at 50% memory usage.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large token-indexed lookup tables provide a compute-decoupled scaling path, but their practical gains are often limited by poor parameter efficiency and rapid memory growth. We attribute these limitations to Zipfian under-training of the long tail, heterogeneous demand across layers, and "slot collapse" that produces redundant embeddings. To address this, we propose X-GRAM, a frequency-aware dynamic token-injection framework. X-GRAM employs hybrid hashing and alias mixing to compress the tail while preserving head capacity, and refines retrieved vectors via normalized SwiGLU ShortConv to extract diverse local n-gram features. These signals are integrated into attention value streams and inter-layer residuals using depth-aware gating, effectively aligning static memory with dynamic context. This design introduces a memory-centric scaling axis that decouples model capacity from FLOPs. Extensive evaluations at the 0.73B and 1.15B scales show that X-GRAM improves average accuracy by as much as 4.4 points over the vanilla backbone and 3.2 points over strong retrieval baselines, while using substantially smaller tables in the 50% configuration. Overall, by decoupling capacity from compute through efficient memory management, X-GRAM offers a scalable and practical paradigm for future memory-augmented architectures. Code aviliable in https://github.com/Longyichen/X-gram.

Problem

• parameter efficiency
• memory growth
• Zipfian distribution
• token embedding
• redundant embeddings

Innovation

• X-GRAM
• memory-efficient embedding
• dynamic token injection
• frequency-aware hashing
• compute-capacity decoupling