Learning and Optimization with 3D Orientations

2025-09-21

AI Summary

Despite the abundance of 3D orientation representations (e.g., quaternions, rotation matrices, Euler angles, axis-angle), their suitability for learning and optimization tasks—such as imitation learning, reinforcement learning, and trajectory optimization—lacks systematic evaluation and unified guidance. Method: We introduce the first comprehensive mathematical framework unifying all major orientation representations, leveraging Lie group and Lie algebra theory to ensure differentiability, singularity-free parameterization, and computational efficiency. Contribution/Results: Through large-scale, cross-task, and cross-algorithm benchmarking, we empirically characterize performance boundaries and trade-offs of each representation under diverse scenarios. We open-source a fully reproducible reference library implementing all representations with efficient forward passes and analytical gradient support. Additionally, we provide task-oriented representation selection guidelines, substantially reducing empirical reliance in algorithm design for orientation modeling.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Machine Learning: Representation Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

There exist numerous ways of representing 3D orientations. Each representation has both limitations and unique features. Choosing the best representation for one task is often a difficult chore, and there exist conflicting opinions on which representation is better suited for a set of family of tasks. Even worse, when dealing with scenarios where we need to learn or optimize functions with orientations as inputs and/or outputs, the set of possibilities (representations, loss functions, etc.) is even larger and it is not easy to decide what is best for each scenario. In this paper, we attempt to a) present clearly, concisely and with unified notation all available representations, and "tricks" related to 3D orientations (including Lie Group algebra), and b) benchmark them in representative scenarios. The first part feels like it is missing from the robotics literature as one has to read many different textbooks and papers in order have a concise and clear understanding of all possibilities, while the benchmark is necessary in order to come up with recommendations based on empirical evidence. More precisely, we experiment with the following settings that attempt to cover most widely used scenarios in robotics: 1) direct optimization, 2) imitation/supervised learning with a neural network controller, 3) reinforcement learning, and 4) trajectory optimization using differential dynamic programming. We finally provide guidelines depending on the scenario, and make available a reference implementation of all the orientation math described.

Problem

• Choosing optimal 3D orientation representations for learning and optimization tasks
• Benchmarking orientation representations across robotics scenarios like optimization and learning
• Providing clear guidelines and implementation for 3D orientation mathematics

Innovation

• Benchmarking 3D orientation representations in robotics
• Providing unified notation for all orientation representations
• Offering guidelines and reference implementation for scenarios