HORST: Composing Optimizer Geometries for Sparse Transformer Training

2026-05-20

AI Summary

Existing optimizers struggle to simultaneously ensure training stability and effectively induce sparsity. This work proposes HORST, an optimizer that introduces hyperbolic mirror maps into the optimization process for the first time, combining non-commutative operators to integrate adaptive optimization with L1-induced sparse geometry. HORST explicitly injects a sparsity bias while maintaining stability on par with AdamW. The resulting framework is modular and geometry-driven, enabling principled sparse optimization. Empirical evaluations across vision and language tasks demonstrate that HORST consistently outperforms AdamW at all sparsity levels, with particularly pronounced gains in high-sparsity regimes.

Abstract

Sparsifying transformers remains a fundamental challenge, as standard optimizers fail to simultaneously encourage sparsity and maintain training stability. Effective adaptive optimizers exhibit an implicit $L_{\infty}$ bias favoring stability, yet, sparsity requires an $L_1$ bias. To integrate sparsity, we propose a composition of optimizer steps, which we cast as non-commutative operators to analyze and combine their optimization geometry in a principled way. This yields HORST (Hyperbolic Operator for Robust Sparse Training), a modular optimizer that inherits stability from adaptive methods while inducing $L_1$ sparsity bias through a hyperbolic mirror map. Our experiments demonstrate its utility for sparse training of transformers on both vision and language tasks. HORST consistently and significantly outperforms AdamW baselines across all sparsity levels, with large gains at higher sparsity.

Problem

• sparsity
• transformer training
• optimization stability
• adaptive optimizers
• L1 bias

Innovation

• sparse training
• optimizer composition
• hyperbolic mirror map
• non-commutative operators
• transformer sparsification