TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing

2026-08-07

AI Summary

This work addresses the parameter explosion bottleneck in existing hypernetwork-based approaches that employ dense, unstructured generators for residual routing, where model size scales prohibitively with the number of routing streams. To mitigate this, the authors model the generator as a trilinear tensor mapping input streams, features, and output streams, and introduce tensor networks for low-rank parameterization. This formulation preserves the dynamic routing capability under manifold constraints while drastically reducing parameter count. By controlling the tensor rank, the method adaptively regulates the dimensionality of the routing subspace, achieving both computational efficiency and interpretability. Experiments on language modeling and commonsense reasoning benchmarks demonstrate that the proposed approach attains state-of-the-art CORE scores with 84% fewer additional parameters (under eight residual streams), significantly outperforming or matching existing methods and establishing a superior trade-off between performance and parameter efficiency.

Abstract

Residual connections rely on a static residual pathway, and are essential for training deep neural networks. Hyper-connections (HC) increase the expressivity of residual routing by incorporating multiple residual streams and learning dynamic information flow, while manifold-constrained (mHC) variants stabilize training through doubly stochastic residual mixing. However, a generator-level bottleneck remains in existing methods: they use dense, unstructured generators for pre-branch aggregation, residual mixing, and post-branch redistribution, which results in parameter count growing rapidly with the number of streams. To address this issue, we propose \underline{\textbf{T}}ensorized \underline{\textbf{E}}fficient \underline{\textbf{M}}anifold-constrained \underline{\textbf{P}}arameterization for \underline{\textbf{E}}xpressive Residual \underline{\textbf{R}}outing (\textbf{TEMPER}), which represents these generators as multi-way tensors over the input-stream, feature, and output-stream modes, and parameterizes them using tensor networks. Such a structured low-rank formulation is shown to preserve token-dependent manifold-constrained routing interface while substantially reducing parameter growth. It also promotes interpretability and intuition, as: i) tensor ranks control the dimensionality of the learned routing subspace, with full ranks recovering dense routing; while ii) the generator approximation errors bound differences in routing logits and, consequently, in the routed-block outputs. Comprehensive experiments show that TEMPER matches or outperforms existing methods across language modeling and commonsense reasoning tasks, while requiring substantially fewer additional parameters. At eight residual streams, TEMPER achieves the best CORE score while using about $84\%$ fewer additional parameters than mHC, thus showing a stronger performance-parameter efficiency trade-off.

Problem

• residual routing
• parameter efficiency
• manifold-constrained
• hyper-connections
• tensorized parameterization

Innovation

• tensor networks
• residual routing
• manifold-constrained
• parameter efficiency
• dynamic information flow