Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction

2026-06-29

AI Summary

This work addresses the challenge of accurately mapping natural language instructions to pixel coordinates in embodied intelligence by proposing a framework that integrates agent-driven data synthesis with multi-module collaborative optimization. The approach leverages mask templates to guide the generation of large-scale candidate data and employs path validation to construct a high-quality training set. It introduces several key innovations: gated cross-block attention (AttnRes), vision-feature-based coordinate perturbation encoding, and an ABC coordinate correction module, enabling joint optimization of semantic understanding, spatial reasoning, and actionability. Evaluated on the PointArena 2026 benchmark, the method achieves an overall accuracy of 77.2% and excels in the Affordance task with 93.9% accuracy, securing second place in the overall ranking.

Abstract

Visual pointing maps a language instruction to pixel co ordinates, a core skill for embodied AI. We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark. The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools; the server inventory contains 55,372 processed out puts, 53,772 de-duplicated sample IDs, and 37,574 train able completed or accepted rows. Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification. Third, two model-side modules address complementary errors: AttnRes adds gated cross-block at tention for steerability, while ABC correction encodes per turbed coordinates with visual features for general coordi nate grounding. Category-aware routing combines comple mentary specialists; local validation used to select experts records 93.9% Affordance, 82.6% Spatial Relation, 78.2% Reasoning, 70.4% Counting, and 63.0% Steerability.

Problem

• visual pointing
• embodied AI
• language-to-pixel grounding
• coordinate prediction
• instruction following

Innovation

• agent-driven data synthesis
• cross-block attention
• iterative correction
• visual pointing
• category-aware routing