SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning

2025-02-07

AI Summary

This study addresses two critical bottlenecks in multi-agent AI systems: overreliance on handcrafted prompts and heuristic rules, and scarcity of domain-specific training data. To this end, we propose a self-improving framework grounded in bootstrapped reasoning. Our method introduces (1) a novel dynamic experience repository that automatically augments training data via failure trajectory refinement; (2) an integrated reasoning paradigm combining self-correction, self-play evolution, and LLM-driven collaborative inference; and (3) trajectory distillation, feedback-guided trajectory selection, and reconstruction. Evaluated on complex reasoning and biomedical question-answering tasks, our approach achieves performance gains of 2.86%–21.88%. It also significantly enhances competitive negotiation capabilities and generates a high-quality, reusable training dataset.

Technology Category

• Multiagent Systems: Multiagent Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Multi-agent AI systems powered by large language models (LLMs) are increasingly applied to solve complex tasks. However, these systems often rely on fragile, manually designed prompts and heuristics, making optimization difficult. A key challenge in optimizing multi-agent systems is acquiring suitable training data for specialized agents. We introduce SiriuS, a self-improving, reasoning-driven optimization framework for multi-agent systems. Central to our approach is the construction of an experience library: a repository of high-quality reasoning trajectories. The library is built by retaining reasoning steps that lead to successful outcomes, providing a robust training set for optimizing multi-agent system. Additionally, we introduce a library augmentation procedure that refines unsuccessful trajectories, further enriching the library. SiriuS boosts performance by 2.86% to 21.88% on reasoning and biomedical QA and enhances agent negotiation in competitive settings. Our results show that SiriuS enhances multi-agent performance while generating reusable data for self-correction and self-play enhancement in the future.

Problem

• Optimize multi-agent systems using self-improving frameworks.
• Acquire high-quality training data for specialized agents.
• Enhance agent performance in reasoning and negotiation tasks.

Innovation

• Self-improving multi-agent systems
• Bootstrapped reasoning framework
• Experience library construction