Botany-Bot: Digital Twin Monitoring of Occluded and Underleaf Plant Structures with Gaussian Splats

2025-10-20

AI Summary

Traditional commercial plant phenotyping systems suffer from fixed-view imaging and occlusion by overlapping leaves, hindering comprehensive acquisition of critical structures—such as stem apices and adaxial/abaxial leaf surfaces. To address this, we propose an active digital twin construction framework for living plants, integrating stereo vision, industrial robotics, and a motorized rotation stage. We design a robot-guided leaf manipulation strategy to reposition occluded organs and, for the first time in plant phenotyping, introduce 3D semantic Gaussian splatting for robust, repeatable imaging and high-fidelity reconstruction of occluded regions. Experiments demonstrate leaf segmentation accuracy of 90.8%, leaf detection accuracy of 86.2%, leaf manipulation success rate of 77.9%, and adaxial/abaxial surface imaging completeness of 77.3%. This work overcomes the limitations of passive sensing, establishing a new paradigm for dynamic, fine-grained structural phenotyping of plants.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Low Level & Physics-based Vision
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Commercial plant phenotyping systems using fixed cameras cannot perceive many plant details due to leaf occlusion. In this paper, we present Botany-Bot, a system for building detailed "annotated digital twins" of living plants using two stereo cameras, a digital turntable inside a lightbox, an industrial robot arm, and 3D segmentated Gaussian Splat models. We also present robot algorithms for manipulating leaves to take high-resolution indexable images of occluded details such as stem buds and the underside/topside of leaves. Results from experiments suggest that Botany-Bot can segment leaves with 90.8% accuracy, detect leaves with 86.2% accuracy, lift/push leaves with 77.9% accuracy, and take detailed overside/underside images with 77.3% accuracy. Code, videos, and datasets are available at https://berkeleyautomation.github.io/Botany-Bot/.

Problem

• Monitoring occluded plant structures using robotic manipulation
• Creating annotated digital twins with 3D Gaussian splat models
• Automating high-resolution imaging of underleaf and stem details

Innovation

• Uses stereo cameras and robot arm for plant monitoring
• Creates 3D digital twins with Gaussian Splat models
• Develops algorithms for manipulating leaves to capture occluded details