Cov2Pose: Leveraging Spatial Covariance for Direct Manifold-aware 6-DoF Object Pose Estimation

2026-03-20

AI Summary

Direct regression of 6-degree-of-freedom (6-DoF) object pose from a single RGB image is often hindered by representation discontinuities and insufficient accuracy. This work proposes an end-to-end approach based on spatial covariance pooling, which encodes convolutional features into symmetric positive-definite (SPD) matrices and performs pose regression on the Riemannian manifold. By incorporating covariance pooling, the method preserves second-order spatial statistics, while leveraging the Cholesky decomposition of SPD matrices to construct a continuous, manifold-aware pose representation. Experiments demonstrate that the proposed method outperforms existing direct regression approaches across multiple benchmarks, achieving notably improved robustness and accuracy under occlusion scenarios, thereby validating the efficacy of second-order feature modeling and geometry-aware representations.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Learning with Manifolds
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

In this paper, we address the problem of 6-DoF object pose estimation from a single RGB image. Indirect methods that typically predict intermediate 2D keypoints, followed by a Perspective-n-Point solver, have shown great performance. Direct approaches, which regress the pose in an end-to-end manner, are usually computationally more efficient but less accurate. However, direct heads rely on globally pooled features, ignoring spatial second-order statistics despite their informativeness in pose prediction. They also predict, in most cases, discontinuous pose representations that lack robustness. Herein, we therefore propose a covariance-pooled representation that encodes convolutional feature distributions as a symmetric positive definite (SPD) matrix. Moreover, we propose a novel pose encoding in the form of an SPD matrix via its Cholesky decomposition. Pose is then regressed in an end-to-end manner with a manifold-aware network head, taking into account the Riemannian geometry of SPD matrices. Experiments and ablations consistently demonstrate the relevance of second-order pooling and continuous representations for direct pose regression, including under partial occlusion.

Problem

• 6-DoF object pose estimation
• direct pose regression
• spatial covariance
• manifold-aware representation
• RGB image

Innovation

• covariance pooling
• SPD manifold
• 6-DoF pose estimation
• Cholesky decomposition
• manifold-aware regression