Divide, Deliberate, Decide: A Multi-Agent Framework for Fine-Grained Egocentric Action Recognition

2026-06-16

AI Summary

This work addresses the challenge of fine-grained action recognition in egocentric videos, where visual cues are often subtle and single-model approaches prone to bias. To overcome these limitations, the authors propose a zero-shot, fully local multi-agent framework that operates without fine-tuning or reliance on external APIs. The method segments input videos and leverages heterogeneous vision-language model (VLM) experts that engage in structured deliberation—including mutual querying—followed by decision fusion via Borda count voting. By exploiting decorrelated model priors and a collaborative reasoning protocol, the framework achieves a significant improvement in recognition accuracy under a zero-shot setting. This study demonstrates, for the first time, that heterogeneous multi-agent deliberation can effectively enable fine-grained action understanding without task-specific training, establishing a new performance benchmark in this challenging domain.

Abstract

Fine-grained action recognition in egocentric video is challenging for Vision-Language Models (VLMs): actions often differ only in small visual cues, and a single model tends to be biased toward a subset of these cues. We propose Divide, Deliberate, Decide, a fully-local, zero-shot multi-agent framework in which (i) a VLM orchestrator chunks the video and proposes a top-k candidate label list per segment, (ii) an ensemble of heterogeneous VLM specialists, drawn from different open model families, engages in a structured deliberation that includes a peer-consultation round of questions, and (iii) agent rankings are aggregated with a Borda count and the orchestrator re-ranks its own prediction in light of the specialists' evidence. The entire pipeline runs locally with no fine-tuning. Experiments show that our method positively improves zero-shot action recognition performance over the baseline, highlighting the influence of a heterogeneous deliberation step, showing that the gain stems from decorrelated model priors rather than from additional compute.

Problem

• fine-grained action recognition
• egocentric video
• Vision-Language Models
• model bias
• visual cues

Innovation

• multi-agent framework
• zero-shot action recognition
• heterogeneous VLMs
• structured deliberation
• egocentric video