RynnEC: Bringing MLLMs into Embodied World

2025-08-19

AI Summary

To address the scarcity of 3D-annotated data in embodied cognition and the limited fine-grained physical perception and spatial interaction capabilities of existing multimodal large models, this paper introduces RynnEC—the first video-based multimodal large language model explicitly designed for embodied cognition. Methodologically, RynnEC features: (1) a collaborative region encoder–masked decoder architecture enabling frame-level spatial region modeling and interaction; (2) a region-centric video generation paradigm that powers a fully automated embodied cognition data synthesis pipeline; and (3) the release of RynnEC-Bench, a dedicated benchmark for evaluating embodied cognition capabilities. Built upon general vision–language foundation models, RynnEC achieves state-of-the-art performance on object attribute understanding, instance segmentation, and spatial reasoning tasks. It significantly enhances embodied agents’ fine-grained perception of and interaction with the physical world.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Embodied AI
• Computer Vision: Multi-modal Vision

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

We introduce RynnEC, a video multimodal large language model designed for embodied cognition. Built upon a general-purpose vision-language foundation model, RynnEC incorporates a region encoder and a mask decoder, enabling flexible region-level video interaction. Despite its compact architecture, RynnEC achieves state-of-the-art performance in object property understanding, object segmentation, and spatial reasoning. Conceptually, it offers a region-centric video paradigm for the brain of embodied agents, providing fine-grained perception of the physical world and enabling more precise interactions. To mitigate the scarcity of annotated 3D datasets, we propose an egocentric video based pipeline for generating embodied cognition data. Furthermore, we introduce RynnEC-Bench, a region-centered benchmark for evaluating embodied cognitive capabilities. We anticipate that RynnEC will advance the development of general-purpose cognitive cores for embodied agents and facilitate generalization across diverse embodied tasks. The code, model checkpoints, and benchmark are available at: https://github.com/alibaba-damo-academy/RynnEC

Problem

• Develops a video multimodal model for embodied cognition
• Enables fine-grained region-level video interaction
• Addresses scarcity of annotated 3D embodied datasets

Innovation

• Video multimodal large language model
• Region encoder and mask decoder
• Egocentric video data generation pipeline