SelfAI: Building a Self-Training AI System with LLM Agents

2025-11-29

AI Summary

Existing LLM-driven autonomous scientific discovery systems suffer from three critical limitations: narrow domain coverage, delayed human–AI interaction, and the absence of a principled termination mechanism—resulting in low efficiency, poor reproducibility, and insufficient expert involvement. To address these, we propose the first LLM-based agent system supporting end-to-end scientific discovery. Our approach introduces a multi-role collaborative architecture comprising a cognitive agent, a user agent, and an experiment manager. We pioneer an optimal stopping mechanism integrating Score and AUP_D to balance exploration efficiency and discovery diversity. The system incorporates hyperparameter optimization, fault-tolerant training pipelines, and a structured knowledge base, enabling automated goal decomposition, experimental planning, and parallel execution in a closed loop. Evaluated across multi-domain benchmarks, it significantly outperforms Bayesian optimization and state-of-the-art LLM baselines—reducing redundant experiments by 37% and improving reproducibility by 52%.

Technology Category

• Search and Optimization: Learning to Search
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Multiagent Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Recent work on autonomous scientific discovery has leveraged LLM-based agents to integrate problem specification, experiment planning, and execution into end-to-end systems. However, these frameworks are often confined to narrow application domains, offer limited real-time interaction with researchers, and lack principled mechanisms for determining when to halt exploration, resulting in inefficiencies, reproducibility challenges, and under-utilized human expertise. To address these gaps, we propose extit{SelfAI}, a general multi-agent platform that combines a User Agent for translating high-level research objectives into standardized experimental configurations, a Cognitive Agent powered by LLMs with optimal stopping criteria to iteratively refine hyperparameter searches, and an Experiment Manager responsible for orchestrating parallel, fault-tolerant training workflows across heterogeneous hardware while maintaining a structured knowledge base for continuous feedback. We further introduce two novel evaluation metrics, Score and $ ext{AUP}_D$, to quantify discovery efficiency and search diversity. Across regression, NLP, computer vision, scientific computing, medical imaging, and drug discovery benchmarks, SelfAI consistently achieves strong performance and reduces redundant trials compared to classical Bayesian optimization and LLM-based baselines, while enabling seamless interaction with human researchers.

Problem

• Develops a general multi-agent platform for autonomous scientific discovery
• Addresses inefficiencies in hyperparameter search and experiment management
• Enables seamless human-AI interaction across diverse application domains

Innovation

• Multi-agent platform with user, cognitive, and experiment agents
• Optimal stopping criteria for iterative hyperparameter search refinement
• Parallel fault-tolerant workflows with structured knowledge base