LA-Pose: Latent Action Pretraining Meets Pose Estimation

2026-04-30

AI Summary

This work addresses the heavy reliance of camera pose estimation on large-scale 3D-annotated data by proposing a self-supervised pretraining approach that, for the first time, integrates inverse and forward dynamics models into this task. By learning latent action representations from large-scale unlabeled driving videos and using them as input features to a fine-tuned pose estimator, the method substantially reduces dependence on annotated data. With only a small amount of high-quality 3D annotations, it achieves over a 10% improvement in pose accuracy compared to state-of-the-art feedforward methods on both the Waymo and PandaSet benchmarks, attaining leading performance with an order of magnitude fewer annotations.

Abstract

This paper revisits camera pose estimation through the lens of self-supervised pretraining, focusing on inverse-dynamics pretraining as a scalable alternative to the current trend of fully supervised training with 3D annotations. Concretely, we employ inverse- and forward-dynamics models to learn latent action representations, similar to Genie from large-scale driving videos. Our idea is simple yet effective. Existing methods use latent actions in their original capacity, that is, as action conditioning of world-models or as proxies of robot action parameters in policy networks. Our method, dubbed LA-Pose, repurposes the latent action features as inputs to a camera pose estimator, finetuned on a limited set of high-quality 3D annotations. This formulation enables accurate and generalizable pose prediction while maintaining feed-forward efficiency. Extensive experiments on driving benchmarks show that LA-Pose achieves competitive and even superior performance to state-of-the-art methods while using orders of magnitude less labeled data. Concretely, on the Waymo and PandaSet benchmarks, LA-Pose achieves over 10% higher pose accuracy than recent feed-forward methods. To our knowledge, this work is the first to demonstrate the power of inverse-dynamics self-supervised learning for pose estimation.

Problem

• camera pose estimation
• self-supervised pretraining
• inverse-dynamics
• latent action
• 3D annotations

Innovation

• latent action
• inverse-dynamics pretraining
• camera pose estimation
• self-supervised learning
• 3D annotation efficiency