Enhancing annotations for 5D apple pose estimation through 3D Gaussian Splatting (3DGS)

2025-12-23

AI Summary

In orchard automation, accurate 5D pose estimation of apples is hindered by severe occlusion of discriminative keypoints (e.g., calyx), leading to labor-intensive manual annotation, inter-image annotation inconsistencies, and label scarcity. To address these challenges, we propose the first end-to-end framework integrating 3D Gaussian Splatting (3DGS) reconstruction with pose label propagation. Leveraging sparse human annotations, our method performs scene-level 3D reconstruction and automatically projects pose labels across views via multi-view geometric projection. With only 105 manual annotations, it generates 28,191 high-quality training labels—reducing annotation cost by 99.6%. Our approach achieves F1 scores of 0.927 on real images and 0.970 on rendered images. We further provide the first empirical evidence that occlusion severity significantly degrades positional estimation accuracy. This work pioneers the use of 3D reconstruction to resolve occlusion-induced annotation inconsistency in agricultural computer vision.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: State Estimation
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web

Abstract

Automating tasks in orchards is challenging because of the large amount of variation in the environment and occlusions. One of the challenges is apple pose estimation, where key points, such as the calyx, are often occluded. Recently developed pose estimation methods no longer rely on these key points, but still require them for annotations, making annotating challenging and time-consuming. Due to the abovementioned occlusions, there can be conflicting and missing annotations of the same fruit between different images. Novel 3D reconstruction methods can be used to simplify annotating and enlarge datasets. We propose a novel pipeline consisting of 3D Gaussian Splatting to reconstruct an orchard scene, simplified annotations, automated projection of the annotations to images, and the training and evaluation of a pose estimation method. Using our pipeline, 105 manual annotations were required to obtain 28,191 training labels, a reduction of 99.6%. Experimental results indicated that training with labels of fruits that are $leq95%$ occluded resulted in the best performance, with a neutral F1 score of 0.927 on the original images and 0.970 on the rendered images. Adjusting the size of the training dataset had small effects on the model performance in terms of F1 score and pose estimation accuracy. It was found that the least occluded fruits had the best position estimation, which worsened as the fruits became more occluded. It was also found that the tested pose estimation method was unable to correctly learn the orientation estimation of apples.

Problem

• Automates apple pose estimation in orchards despite occlusions
• Reduces manual annotation effort using 3D reconstruction techniques
• Evaluates performance of pose estimation under varying occlusion levels

Innovation

• 3D Gaussian Splatting reconstructs orchard scenes for annotation
• Automated projection reduces manual annotations by 99.6%
• Training with highly occluded fruits improves pose estimation performance