RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination

2026-07-15

AI Summary

This work addresses the challenge of effectively integrating high-level task reasoning with physical state prediction in embodied intelligence. The authors propose Hy-Embodied-RxBrain, a novel model that, for the first time, couples language-based reasoning and visual imagination within a unified architecture. It jointly generates abstract plans, predicts future world states, and decomposes subgoals through a coherent language–vision planning sequence. Built upon a multimodal hybrid Transformer, the model supports understanding and generation across language, images, and videos, and incorporates an automatic video decomposition mechanism to align textual and visual supervisory signals. An accompanying automated data pipeline and a new benchmark, RxBrain-Bench, validate the approach’s efficacy, demonstrating that the model enables continuous robotic control in real-world settings without requiring large-scale action pretraining.

Abstract

Embodied cognition requires agents to connect high-level task reasoning with the physical states to be achieved. We introduce Hy-Embodied-RxBrain, an embodied cognition foundation model with joint language-visual reasoning and imagination. Unlike vision-language models that emphasize scene understanding and textual decision making, or generative world models that mainly predict future visual states, RxBrain represents embodied plans in a single planning sequence where language and visual imagination play complementary roles. Language provides the abstract structure of a plan, including task decomposition, planning primitives, constraints, temporal order, and decision logic, while visual imagination grounds this structure through world state prediction and joint subgoal planning, associating each planning step with intermediate and final physical states. RxBrain adopts a unified multimodal Mixture-of-Transformers architecture that supports language, image, and video understanding and generation within one model. To train this capability, we build an automatic pipeline that converts embodied videos into joint text-visual planning supervision by decomposing videos into planning steps and aligning them with visual state transitions. We further introduce RxBrain-Bench to evaluate whether models can represent embodied plans through joint textual and visual components rather than separate understanding or generation. Experiments show that RxBrain maintains embodied understanding and generation abilities, and produces plans with coupled textual reasoning, world state prediction, and joint subgoal planning. We also extend RxBrain to continuous robot action generation, where it shows promising real-robot performance without large-scale action-data pretraining. These results provide an initial step toward foundation models for embodied cognition.

Problem

• embodied cognition
• language-visual reasoning
• visual imagination
• foundation model
• embodied planning

Innovation

• embodied cognition
• joint language-visual reasoning
• visual imagination
• multimodal foundation model
• planning sequence