Procedural Generation of Algorithm Discovery Tasks in Machine Learning

2026-03-18

AI Summary

This work addresses critical limitations in current machine learning algorithm discovery—namely, weak evaluation protocols, data contamination, and task homogeneity—by introducing DiscoGen, the first framework to incorporate procedural generation into this domain. DiscoGen dynamically synthesizes millions of diverse, difficulty-controllable tasks by integrating programmable templates from multiple subdomains with tunable parameters. To support rigorous and fair evaluation, the authors also release DiscoBench, a comprehensive benchmark enabling efficient training and assessment of algorithm discovery agents. Experimental results demonstrate the effectiveness of the proposed framework, and its open-source implementation establishes a new paradigm and foundational infrastructure for future research in algorithm discovery.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Evolutionary Learning

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Automating the development of machine learning algorithms has the potential to unlock new breakthroughs. However, our ability to improve and evaluate algorithm discovery systems has thus far been limited by existing task suites. They suffer from many issues, such as: poor evaluation methodologies; data contamination; and containing saturated or very similar problems. Here, we introduce DiscoGen, a procedural generator of algorithm discovery tasks for machine learning, such as developing optimisers for reinforcement learning or loss functions for image classification. Motivated by the success of procedural generation in reinforcement learning, DiscoGen spans millions of tasks of varying difficulty and complexity from a range of machine learning fields. These tasks are specified by a small number of configuration parameters and can be used to optimise algorithm discovery agents (ADAs). We present DiscoBench, a benchmark consisting of a fixed, small subset of DiscoGen tasks for principled evaluation of ADAs. Finally, we propose a number of ambitious, impactful research directions enabled by DiscoGen, in addition to experiments demonstrating its use for prompt optimisation of an ADA. DiscoGen is released open-source at https://github.com/AlexGoldie/discogen.

Problem

• algorithm discovery
• task generation
• evaluation benchmark
• machine learning
• procedural generation

Innovation

• procedural generation
• algorithm discovery
• machine learning
• benchmarking
• automated machine learning