A Unified Optimization Framework for Multiclass Classification with Structured Hyperplane Arrangements

2025-10-06

AI Summary

This work addresses multiclass classification by proposing a unified optimization framework based on structured hyperplane arrangements. Methodologically, it extends SVM principles—maximizing inter-class margins and minimizing misclassification errors—to support nonlinear decision boundaries, while seamlessly integrating geometric structures including classification trees, ℓp-SVMs, and discrete feature selection. Crucially, it is the first to formulate these diverse classifiers within a single mixed-integer programming (MIP) paradigm. To enhance scalability, the framework incorporates kernel tricks and a dynamic clustering–inspired heuristic for efficient large-scale MIP solving. Empirical evaluation on synthetic and UCI benchmark datasets demonstrates that the method achieves classification accuracy comparable to state-of-the-art scikit-learn implementations, yet with significantly faster training times. The approach thus offers both modeling flexibility—through its unifying geometric representation—and computational efficiency—via scalable optimization—making it particularly suitable for complex, high-dimensional classification tasks.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Search and Optimization: Mixed Discrete/Continuous Search
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text

Abstract

In this paper, we propose a new mathematical optimization model for multiclass classification based on arrangements of hyperplanes. Our approach preserves the core support vector machine (SVM) paradigm of maximizing class separation while minimizing misclassification errors, and it is computationally more efficient than a previous formulation. We present a kernel-based extension that allows it to construct nonlinear decision boundaries. Furthermore, we show how the framework can naturally incorporate alternative geometric structures, including classification trees, $ell_p$-SVMs, and models with discrete feature selection. To address large-scale instances, we develop a dynamic clustering matheuristic that leverages the proposed MIP formulation. Extensive computational experiments demonstrate the efficiency of the proposed model and dynamic clustering heuristic, and we report competitive classification performance on both synthetic datasets and real-world benchmarks from the UCI Machine Learning Repository, comparing our method with state-of-the-art implementations available in scikit-learn.

Problem

• Develops optimization model for multiclass classification with hyperplane arrangements
• Extends framework to handle nonlinear boundaries and alternative geometric structures
• Addresses large-scale instances through dynamic clustering matheuristic approach

Innovation

• Unified optimization framework for multiclass classification
• Kernel-based extension for nonlinear decision boundaries
• Dynamic clustering matheuristic for large-scale instances