Transformer Semantic Genetic Programming for Symbolic Regression

2025-01-30

AI Summary

Symbolic regression suffers from inefficient semantic search and program bloat. To address these challenges, we propose Semantic-enhanced Generative Programming (SGP-Transformer), the first approach to employ a generative Transformer as a semantic mutation operator—modeling fine-grained semantic similarity among programs and guiding the generation of compact, high-fidelity offspring within a geometric semantic space. Our method integrates synthetic-data pretraining with a geometric semantic genetic programming framework to enable cross-task semantic generalization. On multiple benchmark problems, SGP-Transformer achieves prediction accuracy comparable to or exceeding that of standard GP, SLIM_GSGP, DSR, and DAE-GP. Moreover, it significantly suppresses program size growth, enhances semantic diversity, and improves global exploration—thereby jointly optimizing accuracy, parsimony, and robustness.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Statistical Relational/Logic Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

In standard genetic programming (stdGP), solutions are varied by modifying their syntax, with uncertain effects on their semantics. Geometric-semantic genetic programming (GSGP), a popular variant of GP, effectively searches the semantic solution space using variation operations based on linear combinations, although it results in significantly larger solutions. This paper presents Transformer Semantic Genetic Programming (TSGP), a novel and flexible semantic approach that uses a generative transformer model as search operator. The transformer is trained on synthetic test problems and learns semantic similarities between solutions. Once the model is trained, it can be used to create offspring solutions with high semantic similarity also for unseen and unknown problems. Experiments on several symbolic regression problems show that TSGP generates solutions with comparable or even significantly better prediction quality than stdGP, SLIM_GSGP, DSR, and DAE-GP. Like SLIM_GSGP, TSGP is able to create new solutions that are semantically similar without creating solutions of large size. An analysis of the search dynamic reveals that the solutions generated by TSGP are semantically more similar than the solutions generated by the benchmark approaches allowing a better exploration of the semantic solution space.

Problem

• Symbolic Regression
• Optimization Algorithm
• Search Efficiency

Innovation

• Transformer Semantics Genetic Programming
• Symbolic Regression
• Semantic Similarity