SAVOR: Skill Affordance Learning from Visuo-Haptic Perception for Robot-Assisted Bite Acquisition

2025-06-03

AI Summary

In robotic assisted feeding, dynamic variations in food physical properties (e.g., hardness, viscosity) and complex utensil–food interactions severely compromise bite acquisition reliability. To address this, we propose a skill manipulability learning framework that uniquely integrates offline-calibrated tool manipulability with online-updated food manipulability: the former is modeled via functional calibration, while the latter employs Bayesian updating—fusing vision-language model (ViLM) priors with multimodal tactile–visual feedback from SAVOR-Net. We further define an interaction-driven skill manipulability metric to enable real-time selection of optimal manipulation skills (e.g., piercing, scooping). Evaluated on 20 single-food items and 10 realistic meals, our method achieves a 13% higher grasping success rate than state-of-the-art category-based approaches, demonstrating significantly improved generalization and robustness.

Technology Category

• Intelligent Robots: Manipulation
• Natural Language Processing: Safety and Robustness
• Computer Vision: Biometrics, Face, Gesture & Pose

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Robot-assisted feeding requires reliable bite acquisition, a challenging task due to the complex interactions between utensils and food with diverse physical properties. These interactions are further complicated by the temporal variability of food properties-for example, steak becomes firm as it cools even during a meal. To address this, we propose SAVOR, a novel approach for learning skill affordances for bite acquisition-how suitable a manipulation skill (e.g., skewering, scooping) is for a given utensil-food interaction. In our formulation, skill affordances arise from the combination of tool affordances (what a utensil can do) and food affordances (what the food allows). Tool affordances are learned offline through calibration, where different utensils interact with a variety of foods to model their functional capabilities. Food affordances are characterized by physical properties such as softness, moisture, and viscosity, initially inferred through commonsense reasoning using a visually-conditioned language model and then dynamically refined through online multi-modal visuo-haptic perception using SAVOR-Net during interaction. Our method integrates these offline and online estimates to predict skill affordances in real time, enabling the robot to select the most appropriate skill for each food item. Evaluated on 20 single-item foods and 10 in-the-wild meals, our approach improves bite acquisition success by 13% over state-of-the-art (SOTA) category-based methods (e.g. use skewer for fruits). These results highlight the importance of modeling interaction-driven skill affordances for generalizable and effective robot-assisted bite acquisition. Website: https://emprise.cs.cornell.edu/savor/

Problem

• Learning skill affordances for robot-assisted bite acquisition
• Modeling utensil-food interactions with diverse physical properties
• Dynamically refining food affordances via visuo-haptic perception

Innovation

• Learns skill affordances from utensil-food interactions
• Combines offline tool and online food affordance learning
• Uses visuo-haptic perception for dynamic property refinement