🤖 AI Summary
This work addresses the challenge of scaling large language models to ultra-long contexts, where the quadratic complexity of full self-attention becomes prohibitive. The authors propose RIS-Kernel, a general-purpose architecture that reduces self-attention complexity from O(N²) to O(N log N) through sparse random geometric sampling, without modifying model weights or requiring GPU acceleration. Notably, this approach pioneers the use of sparse attention as a regularization mechanism, achieving higher inference accuracy even at extremely low densities (e.g., 1%) while circumventing memory bottlenecks. Experiments demonstrate that RIS-Stochastic attains 75.00% accuracy on 32K-context tasks, outperforming dense baselines, and improves over zero-context baselines by 14.06 percentage points on 65K-context tasks—all efficiently executable on standard CPU servers with only 16–128 GB of memory.
📝 Abstract
Full self-attention in large language models scales as O(N^2), which limits long-context document analysis to 65,536 tokens and requires costly GPU clusters. The Reduced Interaction Sampling (RIS) inference engine addresses this constraint as a model-agnostic architecture. Without modifying weights, RIS reduces self-attention complexity to O(N log N) using sparse stochastic geometry that fits within commodity memory limits. We validate RIS on Qwen2-1.5B-Instruct across two regimes. In controlled evaluations at 32,768 tokens (where native dense attention serves as the upper bound), RIS-Stochastic at 1% density and 70 ensemble seeds achieves 75.00% accuracy, outperforming the native dense baseline (71.88%), while RIS-Stochastic at 5% density and 10 seeds matches it (71.88%). This demonstrates that sparse attention acts as a regularizer: low density (1%) over multiple seeds filters out sequence-level noise, whereas higher density (5%) reintroduces distractor noise. Under the tightest budget, RIS-Structural reaches 68.75% accuracy at 1% density with just 10 seeds, recovering 75% of the contextual gap relative to the zero-context floor (59.38%). At 65,536 tokens, where dense attention triggers out-of-memory faults, RIS yields retrieval gains of up to 14.06 percentage points over the zero-context floor (51.56%), which is confirmed as marginally significant under McNemar's paired test (p = 0.078 < 0.10). All evaluations run on commodity, unaccelerated CPU servers (16-128 GB of RAM), demonstrating that long-context LLM inference is feasible on standard academic hardware without GPU acceleration.