Sibyl-AutoResearch: Autonomous Research Needs Self-Evolving Trial-and-Error Harnesses, Not Paper Generators

2026-05-21

AI Summary

Existing autonomous research systems struggle to effectively accumulate and leverage trial-and-error experience, often leading to overstatement of weak evidence, repeated failures, and memory loss. This work proposes the Sibyl-AutoResearch framework, which incorporates a scientific trial-and-error harness mechanism enabling agents to conduct bounded experiments, explicitly store both positive and negative outcomes, and feed this experiential knowledge back into subsequent research phases. The framework introduces two auditable transformation units—trial-to-behavior and trial-to-harness-behavior—and integrates a file-persistence architecture that explicitly models states, roles, memories, gating mechanisms, and artifact trajectories, thereby supporting fully traceable self-evolution and self-repair. Experimental retrospection identified eight high-confidence transformation events (median delay of one round), successfully intercepting or rectifying five common failure modes, including redundant results and outdated data.

Abstract

Autonomous research systems increasingly make the scientific workflow executable: agents can propose ideas, run code, inspect results, and draft papers. But executable workflows do not by themselves produce research judgment. We analyze where current systems lose trial experience: weak evidence becomes prose, pilot signals become broad claims, memory remains textual, and recurring process failures do not change later behavior. We introduce Sibyl-AutoResearch, a self-evolving AutoResearch framework built around Scientific Trial-and-Error Harnesses. A harness lets agents run bounded trials, preserve positive and negative outcomes, and route lessons into later planning, validation, claim scope, scheduling, critique, writing, and harness repair. We formalize this through two auditable conversion units: trial-to-behavior conversion, which links trial signals to later research actions, and trial-to-harness-behavior conversion, which links recurring process failures to system updates. We implement the framework in SIBYL, a file-backed autonomous research system that exposes the state, roles, memory, gates, and artifact traces needed to inspect these conversion paths. A retrospective audit identifies eight high-confidence conversion events, with a median latency of one iteration and a maximum latency of three iterations. A recovered-failure registry further shows how five naturally occurring failure classes, including duplicate results, stale numbers, and unsupported statistics, were blocked, downgraded, or routed into later repair. These traces do not establish a comparative performance claim; they show that the proposed conversion units are recoverable from realistic autonomous-research workspaces. The SIBYL framework and system are available at https://github.com/Sibyl-Research-Team/AutoResearch-SibylSystem.

Problem

• autonomous research
• trial-and-error
• research judgment
• self-evolving systems
• scientific workflow

Innovation

• Scientific Trial-and-Error Harnesses
• trial-to-behavior conversion
• self-evolving autonomous research
• auditable conversion units
• failure registry