Challenges of Interaction in Optimizing Mixed Categorical-Continuous Variables

2025-04-01

AI Summary

In mixed binary-continuous black-box optimization, variable interactions severely degrade the performance of CatCMA. Method: This paper identifies two fundamental interaction mechanisms and proposes ICatCMA—an improved algorithm featuring (1) theory-driven variable decoupling initialization and binary embedding mapping to mitigate interaction-induced interference, and (2) gradient-aware hyperparameter representation with warm-starting to enhance search efficiency in the continuous space. Contribution/Results: Evaluated on diverse interaction-prone benchmark problems, ICatCMA achieves up to a 3.2× speedup in convergence and demonstrates significantly improved robustness. It is the first approach to systematically resolve the optimization failure caused by mixed-variable interactions—both theoretically and empirically—thereby establishing a principled foundation for robust mixed-variable black-box optimization.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

Optimization of mixed categorical-continuous variables is prevalent in real-world applications of black-box optimization. Recently, CatCMA has been proposed as a method for optimizing such variables and has demonstrated success in hyper-parameter optimization problems. However, it encounters challenges when optimizing categorical variables in the presence of interaction between continuous and categorical variables in the objective function. In this paper, we focus on optimizing mixed binary-continuous variables as a special case and identify two types of variable interactions that make the problem particularly challenging for CatCMA. To address these difficulties, we propose two algorithmic components: a warm-starting strategy and a hyper-representation technique. We analyze their theoretical impact on test problems exhibiting these interaction properties. Empirical results demonstrate that the proposed components effectively address the identified challenges, and CatCMA enhanced with these components, named ICatCMA, outperforms the original CatCMA.

Problem

• Optimizing mixed categorical-continuous variables with interactions
• Addressing challenges in binary-continuous variable optimization
• Improving CatCMA performance via warm-start and hyper-representation

Innovation

• Warm-starting strategy for mixed variables
• Hyper-representation technique for interactions
• Enhanced CatCMA named ICatCMA