Symbolic regression via MDLformer-guided search: from minimizing prediction error to minimizing description length

2024-11-06

arXiv.org

AI Summary

In symbolic regression, non-monotonic prediction error undermines convergence stability and formula recovery rates of conventional methods. To address this, we propose an MDL-driven paradigm that replaces error minimization with Minimum Description Length (MDL) minimization as the search objective—ensuring strict monotonic decrease of the objective as approximations approach the true underlying formula. We design MDLformer, a generalizable Transformer-based neural network for robust MDL estimation of symbolic expressions, and introduce SR4MDL, a gradient-free search algorithm integrating large-scale synthetic data pretraining and symbolic encoding techniques. Evaluated on 133 benchmark problems, our method recovers approximately 50 ground-truth formulas—surpassing state-of-the-art approaches by 43.92% in recovery rate. Furthermore, it demonstrates strong generalization on 122 unseen black-box tasks, validating its robustness beyond synthetic settings.

Technology Category

• Machine Learning: Neuro-Symbolic Learning
• Search and Optimization: Sampling/Simulation-based Search
• Cognitive Modeling & Cognitive Systems: Symbolic Representations

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Symbolic regression, a task discovering the formula best fitting the given data, is typically based on the heuristical search. These methods usually update candidate formulas to obtain new ones with lower prediction errors iteratively.However, since formulas with similar function shapes may have completely different symbolic forms, the prediction error does not decrease monotonously as the search approaches the target formula, causing the low recovery rate of existing methods. To solve this problem, we propose a novel search objective based on the minimum description length, which reflects the distance from the target and decreases monotonically as the search approaches the correct form of the target formula. To estimate the minimum description length of any input data, we design a neural network, MDLformer, which enables robust and scalable estimation through large-scale training. With the MDLformer's output as the search objective, we implement a symbolic regression method, SR4MDL, that can effectively recover the correct mathematical form of the formula. Extensive experiments illustrate its excellent performance in recovering formulas from data. Our method successfully recovers around 50 formulas across two benchmark datasets comprising 133 problems, outperforming state-of-the-art methods by 43.92%. Experiments on 122 unseen black-box problems further demonstrate its generalization performance. We release our code at https://github.com/tsinghua-fib-lab/SR4MDL .

Problem

• Improves symbolic regression by minimizing description length.
• Addresses low recovery rates in formula discovery methods.
• Enhances generalization and accuracy in mathematical formula recovery.

Innovation

• Uses MDLformer for minimum description length estimation
• Implements SR4MDL for symbolic regression
• Achieves 43.92% better formula recovery rate