A Novel Approach to Tomato Harvesting Using a Hybrid Gripper with Semantic Segmentation and Keypoint Detection

2024-12-21

arXiv.org

AI Summary

Tomato harvesting faces challenges of fruit fragility and complex, dynamic environments, leading to poor recognition robustness and low grasping safety. Method: This paper proposes an autonomous harvesting approach integrating multimodal RGB-D vision perception with a bioinspired hybrid gripper. A semantic segmentation and keypoint detection framework enables precise localization of mature tomatoes under occlusion and varying illumination. The gripper combines a rigid exoskeleton with a flexible negative Poisson’s ratio structure, enabling adjustable grasping force and soft enveloping capability. Scotch-Yoke servo actuation and real-time trajectory planning ensure gentle, stable manipulation. Contribution/Results: Experiments demonstrate 96.2% recognition accuracy across ripeness stages and a 91.7% harvesting success rate—significantly outperforming conventional grippers—establishing a new paradigm for high-precision, low-damage agricultural robotics.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Biometrics, Face, Gesture & Pose
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Current agriculture and farming industries are able to reap advancements in robotics and automation technology to harvest fruits and vegetables using robots with adaptive grasping forces based on the compliance or softness of the fruit or vegetable. A successful operation depends on using a gripper that can adapt to the mechanical properties of the crops. This paper proposes a new robotic harvesting approach for tomato fruit using a novel hybrid gripper with a soft caging effect. It uses its six flexible passive auxetic structures based on fingers with rigid outer exoskeletons for good gripping strength and shape conformability. The gripper is actuated through a scotch-yoke mechanism using a servo motor. To perform tomato picking operations through a gripper, a vision system based on a depth camera and RGB camera implements the fruit identification process. It incorporates deep learning-based keypoint detection of the tomato's pedicel and body for localization in an occluded and variable ambient light environment and semantic segmentation of ripe and unripe tomatoes. In addition, robust trajectory planning of the robotic arm based on input from the vision system and control of robotic gripper movements are carried out for secure tomato handling. The tunable grasping force of the gripper would allow the robotic handling of fruits with a broad range of compliance.

Problem

• Develops an autonomous tomato harvesting system using a hybrid gripper.
• Integrates semantic segmentation and keypoint detection for fruit identification.
• Ensures gentle grasping with force control to prevent damage.

Innovation

• Hybrid gripper with soft fingers and rigid exoskeleton for gentle grasping
• Semantic segmentation and keypoint detection using RGB-D camera and Detectron2
• Closed-loop grasp-force regulation with PID controller and force sensors