Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning

2025-03-18

AI Summary

This work addresses the perception–understanding–action closed loop for AI systems operating in the physical world, proposing the first multimodal large language model (MLLM) framework supporting long-horizon embodied reasoning. Methodologically: (1) it introduces a hierarchical physical ontology—spanning spatial, temporal, and physical dimensions—alongside a two-dimensional embodied ontology to enable cross-modal generalizable reasoning; (2) it designs a four-stage progressive training paradigm, uniquely integrating physics-AI-specific supervised fine-tuning (SFT) and reinforcement learning (RL) deeply into the MLLM training pipeline. Contributions and results: the framework achieves significant gains over baselines on a newly constructed benchmark for physical commonsense and embodied reasoning; SFT and RL stages each yield critical performance improvements; the 8B and 56B models, training code, and evaluation benchmark are open-sourced under the NVIDIA Open Model License.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Multiagent Learning
• Intelligent Robots: Embodied AI

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Physical AI systems need to perceive, understand, and perform complex actions in the physical world. In this paper, we present the Cosmos-Reason1 models that can understand the physical world and generate appropriate embodied decisions (e.g., next step action) in natural language through long chain-of-thought reasoning processes. We begin by defining key capabilities for Physical AI reasoning, with a focus on physical common sense and embodied reasoning. To represent physical common sense, we use a hierarchical ontology that captures fundamental knowledge about space, time, and physics. For embodied reasoning, we rely on a two-dimensional ontology that generalizes across different physical embodiments. Building on these capabilities, we develop two multimodal large language models, Cosmos-Reason1-8B and Cosmos-Reason1-56B. We curate data and train our models in four stages: vision pre-training, general supervised fine-tuning (SFT), Physical AI SFT, and Physical AI reinforcement learning (RL) as the post-training. To evaluate our models, we build comprehensive benchmarks for physical common sense and embodied reasoning according to our ontologies. Evaluation results show that Physical AI SFT and reinforcement learning bring significant improvements. To facilitate the development of Physical AI, we will make our code and pre-trained models available under the NVIDIA Open Model License at https://github.com/nvidia-cosmos/cosmos-reason1.

Problem

• Develop models for physical AI reasoning and decision-making.
• Enhance understanding of physical common sense and embodied reasoning.
• Create benchmarks and train models for physical AI capabilities.

Innovation

• Hierarchical ontology for physical common sense
• Two-dimensional ontology for embodied reasoning
• Multimodal large language models with reinforcement learning