Enhanced Cyclic Coordinate Descent Methods for Elastic Net Penalized Linear Models

2025-10-22

AI Summary

To address the low training efficiency of generalized linear models under elastic-net constraints, this paper proposes an enhanced cyclic coordinate descent (CCD) framework. The method approximates gradients via first-order Taylor expansion, transforming coordinate-wise updates into batched linear algebra operations; it further introduces a tunable parameter $s$ to decouple update dependencies, thereby improving both convergence stability and per-iteration efficiency while mitigating latency and numerical instability inherent in block coordinate descent. Implemented efficiently in C++ using the Eigen library, the approach achieves an average 3× speedup across diverse benchmark datasets, with particularly notable gains in elastic-net regularization path computation. The source code is publicly available.

Technology Category

• Machine Learning: Optimization
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We present a novel enhanced cyclic coordinate descent (ECCD) framework for solving generalized linear models with elastic net constraints that reduces training time in comparison to existing state-of-the-art methods. We redesign the CD method by performing a Taylor expansion around the current iterate to avoid nonlinear operations arising in the gradient computation. By introducing this approximation, we are able to unroll the vector recurrences occurring in the CD method and reformulate the resulting computations into more efficient batched computations. We show empirically that the recurrence can be unrolled by a tunable integer parameter, $s$, such that $s > 1$ yields performance improvements without affecting convergence, whereas $s = 1$ yields the original CD method. A key advantage of ECCD is that it avoids the convergence delay and numerical instability exhibited by block coordinate descent. Finally, we implement our proposed method in C++ using Eigen to accelerate linear algebra computations. Comparison of our method against existing state-of-the-art solvers shows consistent performance improvements of $3 imes$ in average for regularization path variant on diverse benchmark datasets. Our implementation is available at https://github.com/Yixiao-Wang-Stats/ECCD.

Problem

• Developing faster coordinate descent for elastic net linear models
• Replacing nonlinear gradients with efficient Taylor approximations
• Achieving 3x speedup over existing solvers while maintaining convergence

Innovation

• Enhanced cyclic coordinate descent with Taylor expansion
• Unrolled vector recurrences for batched computations
• Tunable parameter s for performance optimization