Evolutionary Feature Engineering for Structured Data

2026-07-01

AI Summary

This study addresses the automatic discovery of efficient and interpretable preprocessing and feature engineering methods for structured data, such as time series and tabular datasets. It proposes an Evolutionary Feature Engineering (EFE) framework that, for the first time, employs large language models as evolutionary search operators to automatically generate Python programs conforming to the standard fit/transform interface. The framework iteratively optimizes these programs by leveraging data context, statistical summaries, and validation performance, enabling end-to-end compatibility with existing machine learning pipelines. On time series tasks, the approach reduces MASE, WQL, and MAE errors by over 3% on average (up to 19%). For tabular data, EFE-Tab produces compact feature sets that achieve competitive accuracy with decision tree models while preserving strong interpretability.

Abstract

Large language models are increasingly used as open-ended search operators in evolutionary optimization. We introduce Evolutionary Feature Engineering (EFE), a framework for using LLM-based evolution to discover preprocessing transformations for structured data. EFE represents transformations as Python programs with a standardized fit/transform interface, allowing them to be inserted directly into existing machine learning pipelines. During evolution, candidate programs are refined using dataset context, summary statistics, and downstream performance feedback on validation set. We instantiate EFE in two settings. For time-series forecasting, EFE-Time learns invertible, dataset-specific normalizations that improve off-the-shelf time-series foundation models. It reduces forecasting errors (MASE, WQL, MAE) 3% or more when averaged across datasets and improvements are as much as 19% on the COVID-Deaths dataset. Notably, these improvements occur with recent TSFMs such as Chronos-2. For tabular prediction, EFE-Tab evolves compact feature programs that add useful interpretable features and remove redundant ones, improving or matching existing LLM-based feature-engineering methods. We found EFE-Tab to be particularly effective on classical decision trees, where small sets of evolved features yield competitive accuracy while preserving interpretability. Overall, EFE demonstrates that LLM-based evolution can improve both accuracy and interpretability when automatically tackling structured data.

Problem

• Feature Engineering
• Structured Data
• Evolutionary Optimization
• Preprocessing Transformations
• Interpretability

Innovation

• Evolutionary Feature Engineering
• Large Language Models
• Structured Data
• Automated Preprocessing
• Interpretable Features