Magma: A Foundation Model for Multimodal AI Agents

2025-02-18

AI Summary

Bridging the gap between digital perception and physical action remains a fundamental challenge in embodied AI, requiring unified modeling of vision-language understanding, spatiotemporal planning, and grounded action execution. Method: This paper introduces Magma—a foundational multimodal embodied agent model that jointly processes digital and physical modalities. It proposes two novel annotation paradigms—Set-of-Mark (SoM) for action-anchoring visual objects and Trace-of-Mark (ToM) for modeling motion trajectories—and designs an end-to-end executable architecture pretrained on heterogeneous multimodal data (images, videos, robot interaction sequences), with SoM/ToM explicitly driving action grounding and spatiotemporal planning. Contribution/Results: Magma is the first model to unify linguistic intelligence with spatiotemporal intelligence within a single architecture. Experiments demonstrate state-of-the-art performance on UI navigation and robotic manipulation tasks, while also surpassing larger baseline models on standard multimodal understanding benchmarks.

Technology Category

• Intelligent Robots: Embodied AI
• Multiagent Systems: Multiagent Planning
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We present Magma, a foundation model that serves multimodal AI agentic tasks in both the digital and physical worlds. Magma is a significant extension of vision-language (VL) models in that it not only retains the VL understanding ability (verbal intelligence) of the latter, but is also equipped with the ability to plan and act in the visual-spatial world (spatial-temporal intelligence) and complete agentic tasks ranging from UI navigation to robot manipulation. To endow the agentic capabilities, Magma is pretrained on large amounts of heterogeneous datasets spanning from images, videos to robotics data, where the actionable visual objects (e.g., clickable buttons in GUI) in images are labeled by Set-of-Mark (SoM) for action grounding, and the object movements (e.g., the trace of human hands or robotic arms) in videos are labeled by Trace-of-Mark (ToM) for action planning. Extensive experiments show that SoM and ToM reach great synergy and facilitate the acquisition of spatial-temporal intelligence for our Magma model, which is fundamental to a wide range of tasks as shown in Fig.1. In particular, Magma creates new state-of-the-art results on UI navigation and robotic manipulation tasks, outperforming previous models that are specifically tailored to these tasks. On image and video-related multimodal tasks, Magma also compares favorably to popular large multimodal models that are trained on much larger datasets. We make our model and code public for reproducibility at https://microsoft.github.io/Magma.

Problem

• Multimodal AI agentic tasks
• Digital and physical worlds integration
• UI navigation and robotic manipulation

Innovation

• Multimodal AI agentic tasks
• Set-of-Mark for action grounding
• Trace-of-Mark for action planning