🤖 AI Summary
This work proposes a brain-inspired neural architecture to address the high computational complexity, large parameter count, and lack of biological plausibility in Transformers when handling long-range dependencies and hierarchical classification tasks. The approach uniquely integrates hyperbolic space embeddings, input-dependent dynamic sparse connectivity, and Hebbian structural learning, modulating connection strengths via geodesic distance decay and incorporating a dual-timescale learning mechanism with fast and slow components. The resulting model reduces computational complexity to O(n·k), achieving 96.5% accuracy on long-range dependency tasks with only 1/15 the parameters of a standard Transformer. On a 20-class hierarchical classification benchmark, it attains 23.8% accuracy using just 41,672 parameters—approximately five times the random baseline—demonstrating substantial gains in both efficiency and biological plausibility.
📝 Abstract
We introduce Resonant Sparse Geometry Networks (RSGN), a brain-inspired architecture with self-organizing sparse hierarchical input-dependent connectivity. Unlike Transformer architectures that employ dense attention mechanisms with O(n^2) computational complexity, RSGN embeds computational nodes in learned hyperbolic space where connection strength decays with geodesic distance, achieving dynamic sparsity that adapts to each input. The architecture operates on two distinct timescales: fast differentiable activation propagation optimized through gradient descent, and slow Hebbian-inspired structural learning for connectivity adaptation through local correlation rules. We provide rigorous mathematical analysis demonstrating that RSGN achieves O(n*k) computational complexity, where k<<n represents the average active neighborhood size. Experimental evaluation on hierarchical classification and long-range dependency tasks demonstrates that RSGN achieves 96.5% accuracy on long-range dependency tasks while using approximately 15x fewer parameters than standard Transformers. On challenging hierarchical classification with 20 classes, RSGN achieves 23.8% accuracy (compared to 5% random baseline) with only 41,672 parameters, nearly 10x fewer than the Transformer baselines which require 403,348 parameters to achieve 30.1% accuracy. Our ablation studies confirm the contribution of each architectural component, with Hebbian learning providing consistent improvements. These results suggest that brain-inspired principles of sparse, geometrically-organized computation offer a promising direction toward more efficient and biologically plausible neural architectures.