Chain Of Interaction Benchmark (COIN): When Reasoning meets Embodied Interaction

2026-04-18

AI Summary

This work addresses the lack of systematic evaluation of embodied agents’ causal reasoning capabilities through continuous interaction in partially observable environments. To this end, we introduce the COIN benchmark, which features COIN-50—a novel embodied task suite comprising 50 everyday interactive tasks—and incorporates hierarchical structures, COIN-Primitive and COIN-Composition, to support multi-level reasoning. Leveraging a low-cost mobile AR teleoperation system, we collect demonstration data and establish baseline systems using behavior cloning, CodeAsPolicy, and vision-language-action (VLA) models. We further propose an evaluation framework that measures execution stability and generalization robustness. Experimental results reveal a significant gap between current methods’ visual understanding and action execution, particularly in multi-step causal reasoning tasks, and provide fine-grained attribution analysis to elucidate failure modes.

Technology Category

• Reasoning under Uncertainty: Causality
• Knowledge Representation and Reasoning: Action, Change, and Causality
• Intelligent Robots: Embodied AI

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Generalist embodied agents must perform interactive, causally-dependent reasoning, continually interacting with the environment, acquiring information, and updating plans to solve long-horizon tasks before they could be adopted in real-life scenarios. For instance, retrieving an apple from a cabinet may require opening multiple doors and drawers before the apple becomes visible and reachable, demanding sequential interaction under partial observability. However, existing benchmarks fail to systematically evaluate this essential capability. We introduce COIN, a benchmark designed to assess interactive reasoning in realistic robotic manipulation through three key contributions. First, we construct COIN-50: 50 interactive tasks in daily scenarios, and create COIN-Primitive required by causally-dependent tasks, and COIN-Composition with mid-term complexity for skill learning and generalization evaluation. Second, we develop a low-cost mobile AR teleoperation system and collect the COIN-Primitive Dataset with 50 demonstrations per primitive task (1,000 in total). Third, we develop systematic evaluation metrics about execution stability and generalization robustness to evaluate CodeAsPolicy, VLA, and language-conditioned H-VLA approaches. Our comprehensive evaluation reveals critical limitations in current methods: models struggle with interactive reasoning tasks due to significant gaps between visual understanding and motor execution. We provide fine-grained analysis of these limitations.

Problem

• embodied interaction
• interactive reasoning
• long-horizon tasks
• partial observability
• causally-dependent tasks

Innovation

• interactive reasoning
• embodied AI
• benchmark
• teleoperation
• causal dependency