Affordance Agent Harness: Verification-Gated Skill Orchestration

2026-05-01

AI Summary

In open-world settings, actionable regions are often difficult to identify due to challenges such as small scale, occlusion, or reflectance, and existing fixed-pipeline methods lack adaptability to instance-level difficulty and error-correction capabilities. This work proposes a closed-loop runtime system that leverages episodic memory to reuse past experiences, dynamically orchestrates heterogeneous skills—including detection, segmentation, and interaction imagination—via a router, and employs a verification gating mechanism to decide whether to commit predictions based on self-consistency, cross-scale stability, and multi-trajectory evidence fusion. The framework enables on-demand skill invocation, error backtracking, and experience reuse under label-free conditions, significantly advancing the accuracy–cost Pareto frontier across multiple affordance benchmarks while reducing both average skill invocation count and latency.

Abstract

Affordance grounding requires identifying where and how an agent should interact in open-world scenes, where actionable regions are often small, occluded, reflective, and visually ambiguous. Recent systems therefore combine multiple skills (e.g., detection, segmentation, interaction-imagination), yet most orchestrate them with fixed pipelines that are poorly matched to per-instance difficulty, offer limited targeted recovery from intermediate errors, and fail to reuse experience from recurring objects. These failures expose a systems problem: test-time grounding must acquire the right evidence, decide whether that evidence is reliable enough to commit, and do so under bounded inference cost without access to labels. We propose Affordance Agent Harness, a closed-loop runtime that unifies heterogeneous skills with an evidence store and cost control, retrieves episodic memories to provide priors for recurring categories, and employs a Router to adaptively select and parameterize skills. An affordance-specific Verifier then gates commitments using self-consistency, cross-scale stability, and evidence sufficiency, triggering targeted retries before a final judge fuses accumulated evidence and trajectories into the prediction. Experiments on multiple affordance benchmarks and difficulty-controlled subsets show a stronger accuracy-cost Pareto frontier than fixed-pipeline baselines, improving grounding quality while reducing average skill calls and latency. Project page: https://tenplusgood.github.io/a-harness-page/.

Problem

• affordance grounding
• open-world scenes
• skill orchestration
• evidence reliability
• bounded inference

Innovation

• affordance grounding
• skill orchestration
• evidence verification
• adaptive routing
• episodic memory