Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design

2026-05-15

AI Summary

This work proposes a dual-framework approach enabling large language models to autonomously design novel neural architectures that surpass conventional Transformers through recursive self-improvement. The framework comprises AIRA-Compose for high-level architecture search and AIRA-Design for low-level mechanism implementation, orchestrated via multi-agent collaboration to automatically discover high-performance network structures and attention mechanisms within constrained time budgets. Integrating 11 architecture-search agents and 20 mechanism-design agents, the system leverages extrapolation training across models ranging from millions to billions of parameters and generates attention mechanisms tailored for long-range dependencies. The resulting 1B-parameter pretrained model achieves up to a 3.8% accuracy gain on downstream tasks, demonstrates 71% higher scaling efficiency than Llama 3.2, and matches or exceeds human-designed state-of-the-art performance on the Long Range Arena and Autoresearch benchmarks.

Abstract

Toward recursive self-improvement, we investigate LLM agents autonomously designing foundation models beyond standard Transformers. We introduce a dual-framework approach: AIRA-Compose for high-level architecture search, and AIRA-Design for low-level mechanistic implementation. AIRA-Compose uses 11 agents to explore fundamental computational primitives under a 24-hour budget. Agents evaluate million-parameter candidates, extrapolating top designs to 350M, 1B, and 3B scales. This yields 14 architectures across two families: AIRAformers (Transformer-based) and AIRAhybrids (Transformer-Mamba). Pre-trained at 1B scale, these consistently outperform Llama 3.2 and Composer-found baselines. On downstream tasks, AIRAformer-D and AIRAhybrid-D improve accuracy by 2.4% and 3.8% over Llama 3.2. Furthermore, AIRA-Compose finds models with highly efficient scaling frontiers: AIRAformer-C scales 54% and 71% faster than Llama 3.2 and Composer's best Transformer, while AIRAhybrid-C outscales Nemotron-2 by 23% and Composer's best hybrid by 37%. AIRA-Design tasks 20 agents with writing novel attention mechanisms for long-range dependencies and high-performing training scripts. On the Long Range Arena benchmark, agent-designed architectures reach within 2.3% and 2.6% of human state-of-the-art on document matching and text classification. On the Autoresearch benchmark, Greedy Opus 4.5 achieves 0.968 validation bits-per-byte under a fixed time budget, surpassing the published minimum. Together, these frameworks show AI agents can autonomously discover architectures and algorithmic optimizations matching or surpassing hand-designed baselines. This establishes a powerful paradigm for discovering next-generation foundation models, marking a clear step toward recursive self-improvement.

Problem

• neural architecture search
• foundation models
• recursive self-improvement
• LLM agents
• automated design

Innovation

• Agentic Architecture Search
• Recursive Self-Improvement
• Foundation Model Discovery
• Transformer-Mamba Hybrid
• Autonomous AI Design