SemanticFeels: Semantic Labeling during In-Hand Manipulation

2026-02-15

AI Summary

Current robotic in-hand manipulation systems struggle to simultaneously perceive an object’s geometric shape and semantic attributes—such as material composition—limiting their adaptive capabilities. This work proposes a novel approach that integrates high-resolution tactile sensing with neural implicit representations, uniquely combining the Digit tactile sensor with an enhanced signed distance field (SDF) network. By leveraging a fine-tuned EfficientNet-B0 to process tactile images, the method jointly reconstructs object geometry and predicts continuous material regions during manipulation. Evaluated on both single- and multi-material objects, the approach achieves an average material matching accuracy of 79.87%, significantly advancing fine-grained, high-fidelity semantic modeling of real-world material distributions.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Adversarial Attacks & Robustness
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

As robots become increasingly integrated into everyday tasks, their ability to perceive both the shape and properties of objects during in-hand manipulation becomes critical for adaptive and intelligent behavior. We present SemanticFeels, an extension of the NeuralFeels framework that integrates semantic labeling with neural implicit shape representation, from vision and touch. To illustrate its application, we focus on material classification: high-resolution Digit tactile readings are processed by a fine-tuned EfficientNet-B0 convolutional neural network (CNN) to generate local material predictions, which are then embedded into an augmented signed distance field (SDF) network that jointly predicts geometry and continuous material regions. Experimental results show that the system achieves a high correspondence between predicted and actual materials on both single- and multi-material objects, with an average matching accuracy of 79.87% across multiple manipulation trials on a multi-material object.

Problem

• semantic labeling
• in-hand manipulation
• material classification
• tactile perception
• shape representation

Innovation

• semantic labeling
• in-hand manipulation
• neural implicit representation
• tactile sensing
• signed distance field