Contrastive Concept-Tree Search for LLM-Assisted Algorithm Discovery

2026-02-03

AI Summary

This work addresses the inefficiency of program search in large language model (LLM)-assisted algorithm discovery by proposing Contrastive Concept Tree Search (CCTS). CCTS introduces an explicit hierarchical concept representation, extracting concept trees from generated programs and leveraging contrastive learning to compute likelihood ratio scores between high- and low-performance solutions. These scores reweight parent selection during search, prioritizing effective concept combinations while avoiding misleading ones. Unlike approaches relying on LLMs’ implicit algorithmic lineage, CCTS explicitly exploits “concepts to avoid,” enhancing both search efficiency and interpretability. Evaluated on a benchmark of Erdős-style combinatorial problems, CCTS significantly improves search performance, produces task-relevant interpretable concept trees, and demonstrates superior search dynamics in synthetic environments.

Technology Category

• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language Model (LLM)-assisted algorithm discovery is an iterative, black-box optimization process over programs to approximatively solve a target task, where an LLM proposes candidate programs and an external evaluator provides task feedback. Despite intense recent research on the topic and promising results, how can the LLM internal representation of the space of possible programs be maximally exploited to improve performance is an open question. Here, we introduce Contrastive Concept-Tree Search (CCTS), which extracts a hierarchical concept representation from the generated programs and learns a contrastive concept model that guides parent selection. By reweighting parents using a likelihood-ratio score between high- and low-performing solutions, CCTS biases search toward useful concept combinations and away from misleading ones, providing guidance through an explicit concept hierarchy rather than the algorithm lineage constructed by the LLM. We show that CCTS improves search efficiency over fitness-based baselines and produces interpretable, task-specific concept trees across a benchmark of open Erd\H{o}s-type combinatorics problems. Our analysis indicates that the gains are driven largely by learning which concepts to avoid. We further validate these findings in a controlled synthetic algorithm-discovery environment, which reproduces qualitatively the search dynamics observed with the LLMs.

Problem

• LLM-assisted algorithm discovery
• program space representation
• search efficiency
• concept hierarchy
• black-box optimization

Innovation

• Contrastive Concept-Tree Search
• LLM-assisted algorithm discovery
• concept hierarchy
• program synthesis
• contrastive learning