T-REX: Vision-Based System for Autonomous Leaf Detection and Grasp Estimation

2025-05-03

AI Summary

Autonomous leaf sampling in controlled-environment agriculture (CEA) remains challenging due to complex plant geometry, occlusions, and the need for non-destructive tissue acquisition. Method: This paper proposes a vision-guided robotic system integrating real-time semantic segmentation (YOLOv8) and dense depth estimation (RAFT-Stereo) for 3D leaf localization; introduces a novel leaf selection strategy balancing visibility, occlusion level, and proximity constraints; develops a grasp-point estimation algorithm based on surface planarity, top-down accessibility, and edge margin; and implements ROS-based motion control with a custom micro-needle end-effector enabling minimally invasive puncture grasping. Contribution/Results: Evaluated on multi-pose artificial plants, the system achieves a 66.6% stable grasping success rate, demonstrating—for the first time—the feasibility of high-precision, autonomous, in vivo leaf sampling under severe occlusion conditions in CEA settings.

Technology Category

• Intelligent Robots: State Estimation
• Computer Vision: Vision for Robotics & Autonomous Driving
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Search and Retrieval-Augmented AI: Agentic search

Abstract

T-Rex (The Robot for Extracting Leaf Samples) is a gantry-based robotic system developed for autonomous leaf localization, selection, and grasping in greenhouse environments. The system integrates a 6-degree-of-freedom manipulator with a stereo vision pipeline to identify and interact with target leaves. YOLOv8 is used for real-time leaf segmentation, and RAFT-Stereo provides dense depth maps, allowing the reconstruction of 3D leaf masks. These observations are processed through a leaf grasping algorithm that selects the optimal leaf based on clutter, visibility, and distance, and determines a grasp point by analyzing local surface flatness, top-down approachability, and margin from edges. The selected grasp point guides a trajectory executed by ROS-based motion controllers, driving a custom microneedle-equipped end-effector to clamp the leaf and simulate tissue sampling. Experiments conducted with artificial plants under varied poses demonstrate that the T-Rex system can consistently detect, plan, and perform physical interactions with plant-like targets, achieving a grasp success rate of 66.6%. This paper presents the system architecture, implementation, and testing of T-Rex as a step toward plant sampling automation in Controlled Environment Agriculture (CEA).

Problem

• Autonomous leaf detection and grasping in greenhouses
• Real-time 3D leaf reconstruction for optimal grasp selection
• Robotic system for plant sampling automation in CEA

Innovation

• Uses YOLOv8 for real-time leaf segmentation
• Employs RAFT-Stereo for dense 3D depth maps
• Integrates ROS-based motion for precise grasping