Category-Level 6D Object Pose Estimation in Agricultural Settings Using a Lattice-Deformation Framework and Diffusion-Augmented Synthetic Data

2025-05-30

AI Summary

Accurate category-level 6D pose estimation of highly deformable fruits and vegetables remains challenging in agricultural settings due to large intra-class shape variations, absence of instance-level CAD models, and limited availability of depth sensors. Method: This paper proposes the first RGB-only category-level 6D pose estimation framework tailored for highly deformable objects. It (1) introduces a lattice deformation network that jointly optimizes both the 6D pose and geometric shape of a category-level deformable mesh; and (2) pioneers the use of Stable Diffusion for texture enhancement of agricultural images to mitigate real-to-synthetic domain shift. Contribution/Results: Evaluated on a multi-morphology banana benchmark, our method significantly outperforms state-of-the-art approaches (e.g., MegaPose), achieving high-accuracy, geometry-adaptive pose estimation. Furthermore, it has been successfully deployed on a real-world agricultural robotic grasping system.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Agentic search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Accurate 6D object pose estimation is essential for robotic grasping and manipulation, particularly in agriculture, where fruits and vegetables exhibit high intra-class variability in shape, size, and texture. The vast majority of existing methods rely on instance-specific CAD models or require depth sensors to resolve geometric ambiguities, making them impractical for real-world agricultural applications. In this work, we introduce PLANTPose, a novel framework for category-level 6D pose estimation that operates purely on RGB input. PLANTPose predicts both the 6D pose and deformation parameters relative to a base mesh, allowing a single category-level CAD model to adapt to unseen instances. This enables accurate pose estimation across varying shapes without relying on instance-specific data. To enhance realism and improve generalization, we also leverage Stable Diffusion to refine synthetic training images with realistic texturing, mimicking variations due to ripeness and environmental factors and bridging the domain gap between synthetic data and the real world. Our evaluations on a challenging benchmark that includes bananas of various shapes, sizes, and ripeness status demonstrate the effectiveness of our framework in handling large intraclass variations while maintaining accurate 6D pose predictions, significantly outperforming the state-of-the-art RGB-based approach MegaPose.

Problem

• Estimating 6D object poses in agriculture with high variability
• Overcoming reliance on instance-specific CAD models or depth sensors
• Bridging domain gap between synthetic and real-world agricultural data

Innovation

• Lattice-deformation framework for pose estimation
• Diffusion-augmented synthetic data for realism
• RGB-only input for agricultural applications