FactCheck: Feasibility-aware Long-term Action Anticipation with Multi-agent Collaboration

2026-06-10

AI Summary

Existing long-term action prediction methods often generate implausible actions—such as object hallucinations, functional violations, or state inconsistencies—due to the absence of explicit verification against physical environment constraints. This work proposes FactCheck, a novel framework that introduces, for the first time, a multi-agent closed-loop verification mechanism. The approach decouples action prediction into three stages: observation, planning, and verification, and incorporates a dual-form structured memory comprising action summaries and action graphs to integrate high-level intent with low-level environmental states. FactCheck features an action recognizer, a conditional generator, and an action-graph-based feasibility verifier. Evaluated on EPIC-Kitchens-55 and EGTEA Gaze+, the framework substantially outperforms current state-of-the-art methods, establishing a new paradigm for feasibility-aware long-term action prediction.

Abstract

Long-term action anticipation (LTA) aims to predict an ordered sequence of future verb-noun actions from a partially observed video. While this task serves as the foundation for embodied intelligence, anticipating physically feasible long-term actions remains a critical challenge. Existing methods, which operate in an open-loop manner, often hallucinate non-existent objects, violate object affordances, or disregard object states, as they lack explicit mechanisms to verify action feasibility against the physical environment. To address this, we propose FactCheck, a novel multi-agent collaboration framework that improves feasibility through a closed-loop "Observe-Plan-Verify" mechanism. FactCheck decomposes the complex LTA task into specialized roles: an Observer that recognizes historical actions from video observations and constructs a dual-form structured memory, comprising a History Action Abstract that captures high-level human intentions and environmental status, and a History Action Graph that encodes object states and temporal dependencies; a Planner that generates draft future actions conditioned on both low-level historical actions and high-level History Action Abstract; and a Verifier that rigorously validates the draft against the History Action Graph and refines infeasible actions. Extensive experiments on the EPIC-Kitchens-55 and EGTEA Gaze+ benchmarks demonstrate that FactCheck consistently outperforms state-of-the-art methods. Our work establishes a new paradigm for feasibility-aware long-term action anticipation, effectively closing the loop of action recognition, action prediction and action verification.

Problem

• long-term action anticipation
• action feasibility
• physical environment
• object affordances
• embodied intelligence

Innovation

• long-term action anticipation
• feasibility-aware
• multi-agent collaboration
• closed-loop verification
• structured memory