Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction

2026-09-11

AI Summary

本文提出Embodied-BenchForge框架，通过闭环基准合成和技能编排方法解决自动化生成体感基准时的依赖性问题及缺陷传播问题。

Abstract

Agentic systems offer a promising way to automate embodied benchmark construction, but existing approaches typically cover isolated stages or remain specialized to predefined environments and task families. More importantly, multi-step construction produces dependent intermediate artifacts that are often passed downstream without artifact-specific verification, allowing local defects to propagate into the final benchmark. We present Embodied-BenchForge, an agentic framework that transforms user-specified evaluation intents into complete embodied benchmark artifacts. It formulates construction as Closed-Loop Benchmark Synthesis, integrating forward artifact synthesis with backward verification and repair. Skill-Orchestrated Artifact Synthesis composes typed and reusable skills into executable workflows, while an artifact dependency graph records intermediate outputs and their dependencies. Requirement-Guided Verification and Repair applies artifact-specific contracts throughout construction and uses provenance to trigger local re-execution or upstream rollback when verification fails. Embodied-BenchForge constructs six benchmarks covering diverse embodied scenarios in the Offline EQA Track, together with one interactive benchmark containing 220 executable tasks in the Interactive Embodied Track. Evaluations of representative MLLMs and embodied agents show that the benchmarks distinguish model capabilities in both observation-based understanding and closed-loop execution. Quality assessment and ablations validate benchmark quality and the effectiveness of verification and repair, while repair and skill-reuse analyses demonstrate efficient localized recovery and cross-benchmark reusability.

Problem

• Agentic Systems
• Benchmark Construction
• Closed-Loop
• Verification and Repair
• Artifact Dependency

Innovation

• Closed-Loop Benchmark Synthesis
• Skill-Orchestrated Artifact Synthesis
• Requirement-Guided Verification and Repair