CatCMA with Margin: Stochastic Optimization for Continuous, Integer, and Categorical Variables

2025-04-10

AI Summary

Mixed-variable black-box optimization (MV-BBO) faces challenges in jointly optimizing continuous, integer, and categorical variables, particularly due to inadequate modeling of inter-variable dependencies and unstable convergence behavior across variable types. Method: We propose CatCMA—the first unified framework for joint stochastic optimization over all three variable types. It introduces a boundary-aware integer sampling mechanism to stabilize marginal probabilities and improve convergence of continuous variables; moreover, it pioneers the coupling of multivariate Gaussian and categorical distributions to enable adaptive covariance matrix updates and cross-type dependency learning. Results: On diverse mixed-variable benchmark tasks, CatCMA consistently outperforms state-of-the-art Bayesian optimization methods and multiple baselines, demonstrating superior efficiency and generalization—especially under complex, high-order dependency structures.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This study focuses on mixed-variable black-box optimization (MV-BBO), addressing continuous, integer, and categorical variables. Many real-world MV-BBO problems involve dependencies among these different types of variables, requiring efficient methods to optimize them simultaneously. Recently, stochastic optimization methods leveraging the mechanism of the covariance matrix adaptation evolution strategy have shown promising results in mixed-integer or mixed-category optimization. However, such methods cannot handle the three types of variables simultaneously. In this study, we propose CatCMA with Margin (CatCMAwM), a stochastic optimization method for MV-BBO that jointly optimizes continuous, integer, and categorical variables. CatCMAwM is developed by incorporating a novel integer handling into CatCMA, a mixed-category black-box optimization method employing a joint distribution of multivariate Gaussian and categorical distributions. The proposed integer handling is carefully designed by reviewing existing integer handlings and following the design principles of CatCMA. Even when applied to mixed-integer problems, it stabilizes the marginal probability and improves the convergence performance of continuous variables. Numerical experiments show that CatCMAwM effectively handles the three types of variables, outperforming state-of-the-art Bayesian optimization methods and baselines that simply incorporate existing integer handlings into CatCMA.

Problem

• Optimizes mixed continuous, integer, categorical variables jointly
• Addresses dependencies among variable types in black-box optimization
• Improves convergence for mixed-variable problems versus existing methods

Innovation

• Stochastic optimization for mixed-variable black-box problems
• Joint distribution of Gaussian and categorical variables
• Novel integer handling stabilizes marginal probability