BARISTA: A Multi-Task Egocentric Benchmark for Compositional Visual Understanding

2026-05-12

AI Summary

Existing benchmarks struggle to diagnose the causes of model failures in procedural tasks due to their isolated evaluation of subcomponents of scene understanding. This work introduces a densely annotated dataset comprising 185 real-world coffee-making videos, unifying objects, hand–object interactions, temporal relations, and procedural steps into per-frame scene graphs—the first such unified representation in procedural video understanding. To enable multifaceted evaluation, the authors design several zero-shot language-based tasks, including referring expression comprehension, relation extraction, and temporal visual question answering. Experiments reveal substantial performance disparities across different task families, with no single model achieving consistent superiority, thereby demonstrating the benchmark’s diagnostic utility and the inherent challenges in procedural visual understanding.

Abstract

Scene understanding is central to general physical intelligence, and video is a primary modality for capturing both state and temporal dynamics of a scene. Yet understanding physical processes remains difficult, as models must combine object localization, hand-object interactions, relational parsing, temporal reasoning, and step-level procedural inference. Existing benchmarks usually evaluate these capabilities separately, limiting diagnosis of why models fail on procedural tasks. We introduce BARISTA, a densely annotated egocentric dataset and benchmark of 185 real-world coffee-preparation videos covering fully automatic, portafilter-based, and capsule-based workflows. BARISTA provides verified per-frame scene graphs linking persistent object identities to masks, tracks, boxes, attributes, typed relations, hand-object interactions, activities, and process steps. From these graphs, we derive zero-shot language-based tasks spanning phrase grounding, hand-object interaction recognition, referring, activity recognition, relation extraction, and temporal visual question answering. Experiments reveal strong variation across task families and no consistently dominant model family, positioning BARISTA as a challenging diagnostic benchmark for procedural video understanding. Code and dataset available at https://huggingface.co/datasets/ramblr/BARISTA.

Problem

• procedural video understanding
• egocentric vision
• compositional visual understanding
• multi-task benchmark
• scene graph

Innovation

• egocentric video
• compositional visual understanding
• procedural reasoning
• scene graph
• multi-task benchmark