Beyond Point-Attached Semantics: Object-Centric Semantic Fields for Generalizable Manipulation

2026-07-03

AI Summary

This work addresses the lack of stable, viewpoint-invariant 3D semantic understanding of functional object parts—such as handles or openings—in existing robotic manipulation approaches. The authors propose an object-centric continuous semantic field that learns part-aware semantic embeddings from point cloud inputs, enabling queryable representations at arbitrary 3D locations and serving as conditional input for manipulation policies. This formulation overcomes the discreteness and observation dependence of conventional point-level semantics, yielding a continuous, object-level, and viewpoint-invariant semantic representation. Evaluated in both RoboTwin simulation and real-world bimanual robot tasks, the method significantly outperforms baselines using raw point clouds, 2D feature augmentation, or 3D point-level semantics in terms of policy performance and stability in identifying functional parts.

Abstract

Generalizable robot manipulation requires stable 3D understanding of functional object parts, such as handles, tool heads, openings, and graspable regions. Raw point clouds provide geometry but lack explicit part semantics, and their sampled points vary with viewpoint, sensor configuration, and object instance. Existing 2D feature lifting and discrete 3D point-wise features enrich point clouds with semantics, but the resulting features remain attached to observation-dependent samples. We propose an object-centric continuous semantic field that conditions on an object point cloud and reads part-aware semantic embeddings at explicit 3D query locations. The field is trained from part-annotated object models and then frozen to generate semantic point clouds as object-level conditioning for manipulation policies. Experiments on RoboTwin simulation tasks and real-world bimanual object manipulation show that our representation provides more stable functional-part cues and improves policy performance over raw point-cloud, 2D feature lifting, and 3D point-wise feature baselines. Project Page: \href{https://zainzh.github.io/beyond-point-attached-semantics}{https://zainzh.github.io/beyond-point-attached-semantics}.

Problem

• generalizable manipulation
• object-centric semantics
• 3D semantic fields
• point cloud semantics
• functional object parts

Innovation

• object-centric
• semantic field
• generalizable manipulation
• 3D semantic embedding
• functional part perception