Selective Contrastive Learning for Weakly Supervised Affordance Grounding

2025-08-11

AI Summary

Weakly Supervised Affordance Grounding (WSAG) aims to learn to localize affordance-relevant object parts that support specific actions, using only third-person demonstrations without pixel-level annotations. Existing approaches predominantly rely on category-level classification, making them susceptible to intra-class variations unrelated to affordances and hindering precise localization. This paper proposes a selective prototype learning framework coupled with pixel-level contrastive learning to adaptively model functional features at both part- and object-levels. We further integrate CLIP-driven cross-view object association, knowledge distillation, and multi-view consistency optimization to robustly identify action-relevant parts. On standard benchmarks, our method achieves significant improvements in affordance region localization accuracy, advancing the state-of-the-art by +4.2% mAP. To foster reproducibility and future research, we publicly release our code.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Facilitating an entity's interaction with objects requires accurately identifying parts that afford specific actions. Weakly supervised affordance grounding (WSAG) seeks to imitate human learning from third-person demonstrations, where humans intuitively grasp functional parts without needing pixel-level annotations. To achieve this, grounding is typically learned using a shared classifier across images from different perspectives, along with distillation strategies incorporating part discovery process. However, since affordance-relevant parts are not always easily distinguishable, models primarily rely on classification, often focusing on common class-specific patterns that are unrelated to affordance. To address this limitation, we move beyond isolated part-level learning by introducing selective prototypical and pixel contrastive objectives that adaptively learn affordance-relevant cues at both the part and object levels, depending on the granularity of the available information. Initially, we find the action-associated objects in both egocentric (object-focused) and exocentric (third-person example) images by leveraging CLIP. Then, by cross-referencing the discovered objects of complementary views, we excavate the precise part-level affordance clues in each perspective. By consistently learning to distinguish affordance-relevant regions from affordance-irrelevant background context, our approach effectively shifts activation from irrelevant areas toward meaningful affordance cues. Experimental results demonstrate the effectiveness of our method. Codes are available at github.com/hynnsk/SelectiveCL.

Problem

• Identifying functional parts for object interaction affordances
• Overcoming reliance on class-specific irrelevant patterns in WSAG
• Learning affordance-relevant cues at part and object levels

Innovation

• Selective prototypical and pixel contrastive objectives
• Leveraging CLIP for action-associated object discovery
• Cross-referencing complementary views for part-level clues