Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections

2026-03-21

AI Summary

Existing hyperconnection methods are constrained by the Birkhoff polytope, leading to identity degeneration, limited expressivity, and poor parameter efficiency, which disrupt the identity mapping of residual connections and cause training instability. This work proposes spectral-sphere-constrained hyperconnections (sHC), which relocate the feasible set of residual matrices onto the spectral norm sphere. By introducing spectral-sphere geometric constraints for the first time, sHC overcomes the non-negativity limitation inherent in prior approaches, enabling negative-valued entries that facilitate feature subtraction and disentanglement. The method eliminates the need for Sinkhorn iterations or factorial-scale parameters, achieving both parameter efficiency and stable training. sHC substantially enhances model expressiveness and cross-stream interaction flexibility, effectively supporting selective feature diversification and addressing key limitations of current hyperconnection strategies.

Technology Category

• Machine Learning: Learning with Manifolds
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Computer Vision: Representation Learning for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Hyper-Connections (HC) generalize residual connections into multiple streams, employing residual matrices for cross-stream feature mixing to enrich model expressivity. However, unconstrained mixing disrupts the identity mapping property intrinsic to the residual connection, causing unstable training. To address this, Manifold-Constrained Hyper-Connections (mHC) and its variant restrict these matrices to the Birkhoff polytope (doubly stochastic matrices) via Sinkhorn iterations or permutation-based parameterizations. We reveal three limitations of this polytope constraint: (1) identity degeneration, where learned matrices collapse around the identity and diminish cross-stream interactions, (2) an expressivity bottleneck, as the non-negativity constraint prevents subtractive feature disentanglement, and (3) parameterization inefficiencies, manifesting as unstable Sinkhorn iterations or the factorial-scaling overhead of permutation-based parameterizations. To overcome these flaws, we propose Spectral-Sphere-Constrained Hyper-Connections (sHC). By geometrically shifting the feasible set from a rigid polytope to a spectral norm sphere, sHC allows negative entries, unlocking subtractive interactions for selective feature diversification. This shift eliminates unstable Sinkhorn projections and factorial parameterization, enabling expressive, non-degenerate residual matrices while preserving training stability.

Problem

• Hyper-Connections
• Birkhoff polytope
• residual matrices
• expressivity bottleneck
• parameterization inefficiency

Innovation

• Spectral-Sphere Constraint
• Hyper-Connections
• Residual Mixing
• Feature Disentanglement
• Manifold Optimization