Better Understandings and Configurations in MaxSAT Local Search Solvers via Anytime Performance Analysis

2024-03-11

arXiv.org

AI Summary

Existing evaluations of MaxSAT local search solvers overlook dynamic convergence behavior, focusing only on final solution quality and neglecting performance evolution over time. Method: We propose an empirical cumulative distribution function (ECDF)-based analysis method that quantifies solver convergence across multiple instances and time budgets, modeling high-variance anytime performance as an evaluable metric for the first time. We further integrate this ECDF-driven metric into the SMAC hyperparameter optimization framework to enable automated, convergence-aware solver configuration. Contribution/Results: Experimental results demonstrate that our approach yields more robust and efficient parameter configurations compared to conventional tuning strategies optimized solely for final solution quality. The ECDF-guided configuration significantly improves overall solver performance, revealing time-dependent solver strengths and enabling principled assessment of anytime behavior in MaxSAT solving.

Technology Category

• Constraint Satisfaction and Optimization: Solvers and Tools
• Search and Optimization: Mixed Discrete/Continuous Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Though numerous solvers have been proposed for the MaxSAT problem, and the benchmark environment such as MaxSAT Evaluations provides a platform for the comparison of the state-of-the-art solvers, existing assessments were usually evaluated based on the quality, e.g., fitness, of the best-found solutions obtained within a given running time budget. However, concerning solely the final obtained solutions regarding specific time budgets may restrict us from comprehending the behavior of the solvers along the convergence process. This paper demonstrates that Empirical Cumulative Distribution Functions can be used to compare MaxSAT stochastic local search solvers' anytime performance across multiple problem instances and various time budgets. The assessment reveals distinctions in solvers' performance and displays that the (dis)advantages of solvers adjust along different running times. This work also exhibits that the quantitative and high variance assessment of anytime performance can guide machines, i.e., automatic configurators, to search for better parameter settings. Our experimental results show that the hyperparameter optimization tool, i.e., SMAC, can achieve better parameter settings of solvers when using the anytime performance as the cost function, compared to using the metrics based on the fitness of the best-found solutions.

Problem

• Analyze MaxSAT solvers' anytime performance
• Compare solvers across instances and time
• Optimize solver parameters via performance assessment

Innovation

• Using Empirical Cumulative Distribution Functions
• Comparing anytime performance of solvers
• Guiding hyperparameter optimization with anytime performance