The Wilhelm Tell Dataset of Affordance Demonstrations

2025-07-23

AI Summary

Existing functional affordance learning methods predominantly rely on static images or 3D shape annotations, lacking benchmark datasets supporting dynamic video modeling. To address this gap, we introduce the first functional affordance video dataset tailored to household tasks, comprising 7 hours of real-world human manipulation videos captured simultaneously from first-person and third-person viewpoints. The dataset is annotated with fine-grained functional behavior metadata—including task preparation actions, spatial reorganization, and other context-sensitive operations—enabling precise spatiotemporal alignment. It is the first to systematically support learning of temporally grounded, context-dependent affordances directly from multi-view dynamic video. The released large-scale, functionally annotated demonstration dataset significantly advances robotic understanding of functional behaviors, particularly enhancing modeling of prerequisite actions and evolving task contexts. It establishes a new benchmark for training vision-driven functional perception models.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Affordances - i.e. possibilities for action that an environment or objects in it provide - are important for robots operating in human environments to perceive. Existing approaches train such capabilities on annotated static images or shapes. This work presents a novel dataset for affordance learning of common household tasks. Unlike previous approaches, our dataset consists of video sequences demonstrating the tasks from first- and third-person perspectives, along with metadata about the affordances that are manifested in the task, and is aimed towards training perception systems to recognize affordance manifestations. The demonstrations were collected from several participants and in total record about seven hours of human activity. The variety of task performances also allows studying preparatory maneuvers that people may perform for a task, such as how they arrange their task space, which is also relevant for collaborative service robots.

Problem

• Dataset for affordance learning in household tasks
• Video sequences with first- and third-person perspectives
• Training perception systems to recognize affordance manifestations

Innovation

• Video sequences for affordance learning
• First- and third-person perspectives
• Metadata on affordance manifestations