LM-Searcher: Cross-domain Neural Architecture Search with LLMs via Unified Numerical Encoding

2025-09-06

AI Summary

Existing LLM-based neural architecture search (NAS) methods rely on intricate prompt engineering and domain-specific tuning, limiting their generalizability and practicality. To address this, we propose LM-Searcher: the first framework to represent architectures via a unified numerical encoding—NCode—thereby reformulating NAS as an instruction-driven ranking task and eliminating the need for domain adaptation. LM-Searcher further introduces a pruning-guided subspace sampling strategy to efficiently explore high-performing architectural subspaces. Crucially, it requires only standard instruction fine-tuning, without manual priors or task-specific design. We validate LM-Searcher across diverse vision tasks—including image classification, semantic segmentation, and generative modeling—demonstrating strong in-domain optimization capability and cross-domain transfer performance. Experimental results show significant improvements in the universality, robustness, and practical applicability of LLMs for NAS.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Recent progress in Large Language Models (LLMs) has opened new avenues for solving complex optimization problems, including Neural Architecture Search (NAS). However, existing LLM-driven NAS approaches rely heavily on prompt engineering and domain-specific tuning, limiting their practicality and scalability across diverse tasks. In this work, we propose LM-Searcher, a novel framework that leverages LLMs for cross-domain neural architecture optimization without the need for extensive domain-specific adaptation. Central to our approach is NCode, a universal numerical string representation for neural architectures, which enables cross-domain architecture encoding and search. We also reformulate the NAS problem as a ranking task, training LLMs to select high-performing architectures from candidate pools using instruction-tuning samples derived from a novel pruning-based subspace sampling strategy. Our curated dataset, encompassing a wide range of architecture-performance pairs, encourages robust and transferable learning. Comprehensive experiments demonstrate that LM-Searcher achieves competitive performance in both in-domain (e.g., CNNs for image classification) and out-of-domain (e.g., LoRA configurations for segmentation and generation) tasks, establishing a new paradigm for flexible and generalizable LLM-based architecture search. The datasets and models will be released at https://github.com/Ashone3/LM-Searcher.

Problem

• LLM-driven NAS lacks cross-domain adaptability and scalability
• Neural architecture encoding requires domain-specific tuning limitations
• NAS needs flexible generalizable search beyond prompt engineering

Innovation

• Universal numerical encoding for cross-domain architecture representation
• Reformulating NAS as ranking task via instruction-tuning
• Pruning-based subspace sampling for robust transferable learning