DexFruit: Dexterous Manipulation and Gaussian Splatting Inspection of Fruit

2025-08-09

AI Summary

To address the challenges of gentle grasping and coarse-grained damage quantification in autonomous manipulation of delicate fruits (e.g., strawberries, tomatoes, blackberries), this paper proposes a tactile-guided diffusion-policy reinforcement learning framework and FruitSplat—a novel method for 3D damage modeling. The framework integrates optical tactile feedback with diffusion-based policies to enhance grasping robustness; FruitSplat pioneers mapping 2D damage segmentation masks onto high-resolution 3D Gaussian splatting (3DGS) representations, enabling modular and quantifiable 3D damage reconstruction. Technically, the approach unifies optical tactile sensing, semantic segmentation, 3D reconstruction, and 3DGS rendering. Evaluated over 630+ trials, it achieves a 92% grasp success rate, reduces visually detectable bruising by up to 20%, and improves grasping success on challenging fruits by 31%. These advances significantly enhance automation in dexterous handling and precision quality control of fragile horticultural produce.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

DexFruit is a robotic manipulation framework that enables gentle, autonomous handling of fragile fruit and precise evaluation of damage. Many fruits are fragile and prone to bruising, thus requiring humans to manually harvest them with care. In this work, we demonstrate by using optical tactile sensing, autonomous manipulation of fruit with minimal damage can be achieved. We show that our tactile informed diffusion policies outperform baselines in both reduced bruising and pick-and-place success rate across three fruits: strawberries, tomatoes, and blackberries. In addition, we introduce FruitSplat, a novel technique to represent and quantify visual damage in high-resolution 3D representation via 3D Gaussian Splatting (3DGS). Existing metrics for measuring damage lack quantitative rigor or require expensive equipment. With FruitSplat, we distill a 2D strawberry mask as well as a 2D bruise segmentation mask into the 3DGS representation. Furthermore, this representation is modular and general, compatible with any relevant 2D model. Overall, we demonstrate a 92% grasping policy success rate, up to a 20% reduction in visual bruising, and up to an 31% improvement in grasp success rate on challenging fruit compared to our baselines across our three tested fruits. We rigorously evaluate this result with over 630 trials. Please checkout our website at https://dex-fruit.github.io .

Problem

• Autonomous gentle handling of fragile fruits to minimize damage
• Precise evaluation of fruit damage using 3D Gaussian Splatting
• Improving grasp success rate and reducing bruising in robotic fruit harvesting

Innovation

• Optical tactile sensing for gentle fruit handling
• Tactile informed diffusion policies reduce bruising
• 3D Gaussian Splatting for damage quantification