Problem
Research questions and friction points this paper is trying to address.
AstaBench
E2E-Bench-Hard
autonomous agents
research automation
benchmark evaluation
Qiushi Engine on AstaBench E2E-Bench-Hard
🤖 AI Summary
该报告分析了Qiushi Engine v0.8在AstaBench E2E-Bench-Hard上的表现，通过使用DeepSeek模型解决端到端的研究流程自动化问题，实现了10%的全任务完成率。
📝 Abstract
This report analyzes Qiushi Engine v0.8 across all 40 test tasks in AstaBench E2E-Bench-Hard, a benchmark that requires autonomous agents to carry a research question through experimental design, code implementation, actual execution, result analysis, and report delivery. Qiushi Engine is model-configurable; this evaluation selected DeepSeek deepseek-v4pro-preview as the model backend. The official AstaBench leaderboard records a score of 0.816 and an average benchmark cost of USD 15.209 per task, while the full-precision local recomputation is $81.59 \pm 1.87$. Four tasks satisfied every rubric item, yielding a full-task completion rate of 4/40 = 10% -- 7 percentage points above, and about 3.3 times, the approximately 3% best rate reported for AstaBench's official agents. Across 507 required rubric items, 416 were satisfied (82.1%). Official scoring archives and 40 Meta-Trace records show sustained production and verification of reports, code, and experimental artifacts; the principal gaps lie in repeated runs, external dependencies, specified metrics, and ablation studies. The report explains the benchmark, system workflow, aggregate results, representative cases, and limits of interpretation.
Innovation
Methods, ideas, or system contributions that make the work stand out.
autonomous agents
model-configurable
full-task completion rate
AstaBench E2E-Bench-Hard
🔎 Similar Papers
No similar papers found.
