Emergence of a Shared Canonical Object Frame from In-the-Wild Videos

2026-06-29

AI Summary

Existing approaches rely on manually annotated canonical pose labels, which hinders scalability to diverse object categories and instances. This work proposes a self-supervised learning framework that automatically establishes a shared canonical coordinate system across instances using only in-the-wild videos centered on objects and noisy camera poses—without requiring canonical pose annotations or category conditioning. The method leverages a coarse, generic mesh as a geometric bottleneck and integrates multi-view consistency, structure-from-motion (SfM), dense pixel-to-mesh correspondences, and semantic priors to achieve unified coordinate alignment through geometric optimization. Trained on 160,000 in-the-wild video clips, the model attains pose estimation accuracy on category-level benchmarks comparable to that of supervised methods.

Abstract

Comparing object orientations and positions across different instances requires their poses to be expressed in a shared canonical frame. Establishing such frames has traditionally required manual annotation, creating a scaling bottleneck that limits category and instance diversity. We show that a shared canonical frame can instead emerge from self-supervised training on object-centric videos captured in the wild, using only noisy camera poses from Structure-from-Motion. Our key idea is to route all training sequences through a shared geometric bottleneck: a coarse canonical mesh that carries no category-specific detail. By learning dense correspondences from image pixels to this mesh, and estimating per-sequence alignments from noisy SfM geometry, a common canonical frame emerges from multi-view consistency and the semantic priors of the feature extractor, without any canonical pose labels or category conditioning. Trained in a self-supervised manner on 160,000 in-the-wild object videos, our method achieves competitive accuracy on category-level pose estimation benchmarks compared to methods that rely on canonical pose supervision. The code and checkpoint is available on https://github.com/Fischer-Tom/Emergent-Canonical-Frame/.

Problem

• canonical frame
• object pose
• self-supervised learning
• in-the-wild videos
• dense correspondence

Innovation

• canonical frame
• self-supervised learning
• dense correspondence
• Structure-from-Motion
• object pose estimation